2023-10-04T00:19:41.090 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-04T00:19:41.091 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi --archive /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079 --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --owner scheduled_rraja@teuthology -- /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079/orig.config.yaml 2023-10-04T00:19:41.124 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-04T00:19:41.236 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079 branch: wip-ajarr-63009-only-reacquire-kickstart description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7410079' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-ajarr-63009-only-reacquire-kickstart ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ajarr-check-addr-non-zero sha1: 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 owner: scheduled_rraja@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7470 sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-ajarr-check-addr-non-zero suite_path: /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa suite_relpath: qa suite_repo: https://github.com/ajarr/ceph suite_sha1: 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0ecNHL7KfIGkixkKWDByilxad+DFHOUw5+x9jVxBF8gvTOZ1tVfoPG8cDk7ttC44bcMPg9G/SHv/LmT/cUSy8= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-04_00:09:14 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-04T00:19:41.236 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa; will attempt to use it 2023-10-04T00:19:41.237 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks 2023-10-04T00:19:41.237 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-04T00:19:41.239 INFO:teuthology.task.internal:Checking packages... 2023-10-04T00:19:41.264 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad' 2023-10-04T00:19:41.264 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-04T00:19:41.264 INFO:teuthology.packaging:ref: None 2023-10-04T00:19:41.265 INFO:teuthology.packaging:tag: None 2023-10-04T00:19:41.265 INFO:teuthology.packaging:branch: wip-ajarr-63009-only-reacquire-kickstart 2023-10-04T00:19:41.265 INFO:teuthology.packaging:sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-04T00:19:41.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-ajarr-63009-only-reacquire-kickstart 2023-10-04T00:19:41.487 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:19:41.487 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-04T00:19:41.493 INFO:teuthology.task.internal:no buildpackages task found 2023-10-04T00:19:41.493 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-04T00:19:41.502 INFO:teuthology.task.internal:Saving configuration 2023-10-04T00:19:41.521 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-04T00:19:41.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-04T00:19:41.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-04 00:09:38.960880', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID2KUVKZT8PN+lN1l2BLcF+Bc6xriJcIfme1VItWP1d3'} 2023-10-04T00:19:41.552 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-04T00:19:41.557 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-04T00:19:41.557 INFO:teuthology.run_tasks:Running task console_log... 2023-10-04T00:19:41.599 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd6ca88daf0>, signals=[15]) 2023-10-04T00:19:41.599 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-04T00:19:41.605 INFO:teuthology.task.internal:Opening connections... 2023-10-04T00:19:41.605 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:19:41.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:19:41.685 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-04T00:19:41.691 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2023-10-04T00:19:41.791 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2023-10-04T00:19:41.791 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Ubuntu" 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="22.04" 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_CODENAME=jammy 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:ID=ubuntu 2023-10-04T00:19:41.837 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE=debian 2023-10-04T00:19:41.838 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-04T00:19:41.838 INFO:teuthology.orchestra.run.smithi027.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-04T00:19:41.838 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-04T00:19:41.838 INFO:teuthology.orchestra.run.smithi027.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-04T00:19:41.838 INFO:teuthology.orchestra.run.smithi027.stdout:UBUNTU_CODENAME=jammy 2023-10-04T00:19:41.838 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2023-10-04T00:19:41.864 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-04T00:19:41.873 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-04T00:19:41.878 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-04T00:19:41.878 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2023-10-04T00:19:41.884 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-04T00:19:41.889 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-04T00:19:41.889 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2023-10-04T00:19:41.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-04T00:19:41.967 INFO:teuthology.run_tasks:Running task kernel... 2023-10-04T00:19:41.982 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-04T00:19:41.982 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-04T00:19:41.982 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-04T00:19:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T00:19:41.987 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-10-04T00:19:42.033 INFO:teuthology.orchestra.run.smithi027.stdout:5.15.0-56-generic 2023-10-04T00:19:42.033 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.15.0-56-generic 2023-10-04T00:19:42.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get clean 2023-10-04T00:19:42.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get update 2023-10-04T00:19:42.282 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-04T00:19:42.287 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-04T00:19:42.287 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-04T00:19:42.359 INFO:teuthology.orchestra.run.smithi027.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-04T00:19:42.521 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [340 kB] 2023-10-04T00:19:42.590 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [822 kB] 2023-10-04T00:19:42.593 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,054 kB] 2023-10-04T00:19:42.624 INFO:teuthology.orchestra.run.smithi027.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [171 kB] 2023-10-04T00:19:42.629 INFO:teuthology.orchestra.run.smithi027.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-04T00:19:42.630 INFO:teuthology.orchestra.run.smithi027.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [899 kB] 2023-10-04T00:19:42.646 INFO:teuthology.orchestra.run.smithi027.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-04T00:19:42.647 INFO:teuthology.orchestra.run.smithi027.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [145 kB] 2023-10-04T00:19:42.649 INFO:teuthology.orchestra.run.smithi027.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-04T00:19:42.649 INFO:teuthology.orchestra.run.smithi027.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [786 kB] 2023-10-04T00:19:42.659 INFO:teuthology.orchestra.run.smithi027.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-04T00:19:42.662 INFO:teuthology.orchestra.run.smithi027.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [144 kB] 2023-10-04T00:19:42.663 INFO:teuthology.orchestra.run.smithi027.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-04T00:19:42.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-04T00:19:42.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [503 kB] 2023-10-04T00:19:42.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-04T00:19:42.670 INFO:teuthology.orchestra.run.smithi027.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [232 kB] 2023-10-04T00:19:42.673 INFO:teuthology.orchestra.run.smithi027.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-10-04T00:19:42.673 INFO:teuthology.orchestra.run.smithi027.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-04T00:19:42.673 INFO:teuthology.orchestra.run.smithi027.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [967 kB] 2023-10-04T00:19:42.685 INFO:teuthology.orchestra.run.smithi027.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-04T00:19:42.685 INFO:teuthology.orchestra.run.smithi027.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-04T00:19:42.688 INFO:teuthology.orchestra.run.smithi027.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [156 kB] 2023-10-04T00:19:42.689 INFO:teuthology.orchestra.run.smithi027.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-04T00:19:42.689 INFO:teuthology.orchestra.run.smithi027.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [989 kB] 2023-10-04T00:19:42.698 INFO:teuthology.orchestra.run.smithi027.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [658 kB] 2023-10-04T00:19:42.703 INFO:teuthology.orchestra.run.smithi027.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [216 kB] 2023-10-04T00:19:42.704 INFO:teuthology.orchestra.run.smithi027.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.9 kB] 2023-10-04T00:19:42.705 INFO:teuthology.orchestra.run.smithi027.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-04T00:19:42.705 INFO:teuthology.orchestra.run.smithi027.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-04T00:19:42.706 INFO:teuthology.orchestra.run.smithi027.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-04T00:19:42.706 INFO:teuthology.orchestra.run.smithi027.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-04T00:19:42.722 INFO:teuthology.orchestra.run.smithi027.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-04T00:19:42.722 INFO:teuthology.orchestra.run.smithi027.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-04T00:19:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-04T00:19:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-04T00:19:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-04T00:19:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-04T00:19:42.742 INFO:teuthology.orchestra.run.smithi027.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-04T00:19:42.742 INFO:teuthology.orchestra.run.smithi027.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-04T00:19:48.260 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 9,351 kB in 1s (6,583 kB/s) 2023-10-04T00:19:49.507 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:19:49.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-04T00:19:49.589 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:19:49.780 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2023-10-04T00:19:49.780 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2023-10-04T00:19:49.945 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2023-10-04T00:19:49.945 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-04T00:19:49.945 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-04T00:19:49.946 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-04T00:19:49.946 INFO:teuthology.orchestra.run.smithi027.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-04T00:19:49.946 INFO:teuthology.orchestra.run.smithi027.stdout: python2.7-minimal 2023-10-04T00:19:49.946 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-04T00:19:49.947 INFO:teuthology.orchestra.run.smithi027.stdout:The following additional packages will be installed: 2023-10-04T00:19:49.947 INFO:teuthology.orchestra.run.smithi027.stdout: linux-generic linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-04T00:19:49.947 INFO:teuthology.orchestra.run.smithi027.stdout: linux-headers-generic linux-image-5.15.0-86-generic 2023-10-04T00:19:49.947 INFO:teuthology.orchestra.run.smithi027.stdout: linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic 2023-10-04T00:19:49.948 INFO:teuthology.orchestra.run.smithi027.stdout:Suggested packages: 2023-10-04T00:19:49.948 INFO:teuthology.orchestra.run.smithi027.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-04T00:19:49.949 INFO:teuthology.orchestra.run.smithi027.stdout:Recommended packages: 2023-10-04T00:19:49.949 INFO:teuthology.orchestra.run.smithi027.stdout: thermald 2023-10-04T00:19:49.999 INFO:teuthology.orchestra.run.smithi027.stdout:The following NEW packages will be installed: 2023-10-04T00:19:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-04T00:19:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.15.0-86-generic linux-modules-5.15.0-86-generic 2023-10-04T00:19:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: linux-modules-extra-5.15.0-86-generic 2023-10-04T00:19:50.001 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages will be upgraded: 2023-10-04T00:19:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-04T00:19:50.055 INFO:teuthology.orchestra.run.smithi027.stdout:3 upgraded, 5 newly installed, 0 to remove and 266 not upgraded. 2023-10-04T00:19:50.055 INFO:teuthology.orchestra.run.smithi027.stdout:Need to get 113 MB of archives. 2023-10-04T00:19:50.056 INFO:teuthology.orchestra.run.smithi027.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-04T00:19:50.056 INFO:teuthology.orchestra.run.smithi027.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-86-generic amd64 5.15.0-86.96 [22.4 MB] 2023-10-04T00:19:50.518 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-86-generic amd64 5.15.0-86.96 [11.5 MB] 2023-10-04T00:19:50.631 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-86-generic amd64 5.15.0-86.96 [63.9 MB] 2023-10-04T00:19:51.508 INFO:teuthology.orchestra.run.smithi027.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.86.83 [1,702 B] 2023-10-04T00:19:51.509 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.86.83 [2,472 B] 2023-10-04T00:19:51.509 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86 all 5.15.0-86.96 [12.3 MB] 2023-10-04T00:19:51.810 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86-generic amd64 5.15.0-86.96 [2,882 kB] 2023-10-04T00:19:51.836 INFO:teuthology.orchestra.run.smithi027.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.86.83 [2,328 B] 2023-10-04T00:19:52.263 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 113 MB in 2s (61.9 MB/s) 2023-10-04T00:19:52.427 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-modules-5.15.0-86-generic. 2023-10-04T00:19:54.478 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-04T00:19:54.482 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../0-linux-modules-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-04T00:19:54.566 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:19:56.617 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-image-5.15.0-86-generic. 2023-10-04T00:19:56.638 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../1-linux-image-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-04T00:19:56.719 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:19:57.203 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-86-generic. 2023-10-04T00:19:57.224 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-04T00:19:57.261 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:03.668 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../3-linux-generic_5.15.0.86.83_amd64.deb ... 2023-10-04T00:20:03.795 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-04T00:20:04.104 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.86.83_amd64.deb ... 2023-10-04T00:20:04.224 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-image-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-04T00:20:04.519 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-headers-5.15.0-86. 2023-10-04T00:20:04.542 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../5-linux-headers-5.15.0-86_5.15.0-86.96_all.deb ... 2023-10-04T00:20:04.585 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-04T00:20:10.002 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-headers-5.15.0-86-generic. 2023-10-04T00:20:10.040 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../6-linux-headers-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-04T00:20:10.085 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:12.880 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.86.83_amd64.deb ... 2023-10-04T00:20:13.004 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-headers-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-04T00:20:13.339 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-04T00:20:13.484 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:13.594 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-headers-generic (5.15.0.86.83) ... 2023-10-04T00:20:13.721 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:15.086 INFO:teuthology.orchestra.run.smithi027.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-04T00:20:15.086 INFO:teuthology.orchestra.run.smithi027.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-04T00:20:15.086 INFO:teuthology.orchestra.run.smithi027.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-86-generic 2023-10-04T00:20:15.086 INFO:teuthology.orchestra.run.smithi027.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-86-generic 2023-10-04T00:20:15.251 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:18.236 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-image-generic (5.15.0.86.83) ... 2023-10-04T00:20:18.336 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-generic (5.15.0.86.83) ... 2023-10-04T00:20:18.463 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-04T00:20:18.527 INFO:teuthology.orchestra.run.smithi027.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-04T00:20:18.527 INFO:teuthology.orchestra.run.smithi027.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-86-generic 2023-10-04T00:20:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-04T00:20:33.205 INFO:teuthology.orchestra.run.smithi027.stdout:Sourcing file `/etc/default/grub' 2023-10-04T00:20:33.214 INFO:teuthology.orchestra.run.smithi027.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-04T00:20:33.230 INFO:teuthology.orchestra.run.smithi027.stdout:Generating grub configuration file ... 2023-10-04T00:20:33.483 INFO:teuthology.orchestra.run.smithi027.stdout:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-04T00:20:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-04T00:20:33.765 INFO:teuthology.orchestra.run.smithi027.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-04T00:20:33.772 INFO:teuthology.orchestra.run.smithi027.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-04T00:20:33.839 INFO:teuthology.orchestra.run.smithi027.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-04T00:20:33.845 INFO:teuthology.orchestra.run.smithi027.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-04T00:20:34.016 INFO:teuthology.orchestra.run.smithi027.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-04T00:20:34.016 INFO:teuthology.orchestra.run.smithi027.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-04T00:20:34.016 INFO:teuthology.orchestra.run.smithi027.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-04T00:20:34.059 INFO:teuthology.orchestra.run.smithi027.stdout:done 2023-10-04T00:20:34.406 DEBUG:teuthology.orchestra.run.smithi027:> dpkg -s linux-image-generic 2023-10-04T00:20:34.431 INFO:teuthology.orchestra.run.smithi027.stdout:Package: linux-image-generic 2023-10-04T00:20:34.431 INFO:teuthology.orchestra.run.smithi027.stdout:Status: install ok installed 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Priority: optional 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Section: kernel 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed-Size: 20 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Maintainer: Ubuntu Kernel Team 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: amd64 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Source: linux-meta 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Version: 5.15.0.86.83 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-04T00:20:34.432 INFO:teuthology.orchestra.run.smithi027.stdout:Recommends: thermald 2023-10-04T00:20:34.433 INFO:teuthology.orchestra.run.smithi027.stdout:Description: Generic Linux kernel image 2023-10-04T00:20:34.433 INFO:teuthology.orchestra.run.smithi027.stdout: This package will always depend on the latest generic kernel image 2023-10-04T00:20:34.433 INFO:teuthology.orchestra.run.smithi027.stdout: available. 2023-10-04T00:20:34.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-86-generic 2023-10-04T00:20:34.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-04T00:20:34.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-04T00:20:34.433 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-04T00:20:34.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2023-10-04T00:20:34.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get clean 2023-10-04T00:20:34.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get update 2023-10-04T00:20:34.766 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-04T00:20:34.772 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-04T00:20:34.789 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-04T00:20:34.884 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-04T00:20:36.125 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:20:36.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-04T00:20:36.194 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:20:36.362 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2023-10-04T00:20:36.363 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout:linux-image-generic is already the newest version (5.15.0.86.83). 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-04T00:20:36.511 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-04T00:20:36.548 INFO:teuthology.orchestra.run.smithi027.stdout:0 upgraded, 0 newly installed, 0 to remove and 266 not upgraded. 2023-10-04T00:20:36.550 DEBUG:teuthology.orchestra.run.smithi027:> dpkg -s linux-image-generic 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Package: linux-image-generic 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Status: install ok installed 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Priority: optional 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Section: kernel 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Installed-Size: 20 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Maintainer: Ubuntu Kernel Team 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: amd64 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Source: linux-meta 2023-10-04T00:20:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Version: 5.15.0.86.83 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout:Recommends: thermald 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout:Description: Generic Linux kernel image 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout: This package will always depend on the latest generic kernel image 2023-10-04T00:20:36.609 INFO:teuthology.orchestra.run.smithi027.stdout: available. 2023-10-04T00:20:36.609 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-10-04T00:20:36.657 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.5GtUkKd9pZ 2023-10-04T00:20:36.657 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5GtUkKd9pZ 2023-10-04T00:20:36.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.5GtUkKd9pZ 2023-10-04T00:20:36.830 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.5GtUkKd9pZ is 10KB 2023-10-04T00:20:36.843 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.5GtUkKd9pZ 2023-10-04T00:20:36.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-04T00:20:36.908 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:20:36.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-04T00:20:36.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-04T00:20:36.990 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-86-generic 2023-10-04T00:20:36.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo update-grub 2023-10-04T00:20:38.117 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub' 2023-10-04T00:20:38.118 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-04T00:20:38.120 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2023-10-04T00:20:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-04T00:20:38.305 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-04T00:20:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-04T00:20:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-04T00:20:38.577 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-04T00:20:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-04T00:20:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-04T00:20:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-04T00:20:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-04T00:20:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:done 2023-10-04T00:20:38.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2023-10-04T00:21:08.743 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-04T00:21:08.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:21:08.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:21:27.293 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-04T00:21:36.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:21:36.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:21:39.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-04T00:21:51.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:21:51.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:21:54.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-04T00:22:09.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:22:09.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:22:12.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-04T00:22:30.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:22:30.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:22:48.954 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-04T00:23:09.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:23:09.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:23:10.216 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-10-04T00:23:10.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:23:10.877 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-86-generic"... 2023-10-04T00:23:10.878 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-10-04T00:23:10.925 INFO:teuthology.orchestra.run.smithi027.stdout:5.15.0-86-generic 2023-10-04T00:23:10.925 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-86-generic vs 5.15.0-86-generic 2023-10-04T00:23:10.925 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-04T00:23:10.925 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-04T00:23:11.926 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-04T00:23:11.927 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-04T00:23:12.011 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2023-10-04T00:23:12.035 DEBUG:teuthology.parallel:result is None 2023-10-04T00:23:12.036 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-04T00:23:12.044 INFO:teuthology.task.internal:Creating test directory... 2023-10-04T00:23:12.044 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-04T00:23:12.050 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-04T00:23:12.058 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-04T00:23:12.064 INFO:teuthology.task.internal:Creating archive directory... 2023-10-04T00:23:12.065 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-04T00:23:12.109 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-04T00:23:12.115 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-04T00:23:12.116 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-04T00:23:12.166 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-04T00:23:12.174 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-04T00:23:12.175 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-04T00:23:12.183 INFO:teuthology.task.internal:Configuring sudo... 2023-10-04T00:23:12.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-04T00:23:12.229 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-04T00:23:12.237 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-04T00:23:12.238 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-04T00:23:12.281 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-04T00:23:12.335 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-04T00:23:12.381 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:23:12.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-04T00:23:12.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2023-10-04T00:23:12.556 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-04T00:23:12.564 INFO:teuthology.task.internal:Starting timer... 2023-10-04T00:23:12.564 INFO:teuthology.run_tasks:Running task pcp... 2023-10-04T00:23:12.576 INFO:teuthology.run_tasks:Running task selinux... 2023-10-04T00:23:12.587 DEBUG:teuthology.task.selinux:Excluding smithi027: OS 'ubuntu' does not support SELinux 2023-10-04T00:23:12.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-04T00:23:12.588 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-04T00:23:12.588 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-04T00:23:12.588 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-04T00:23:12.599 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-04T00:23:12.606 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-04T00:23:12.744 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-04T00:23:12.762 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-04T00:23:12.763 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-04T00:30:58.210 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com')] 2023-10-04T00:30:58.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:30:58.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-04T00:30:58.290 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-10-04T00:30:58.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-04T00:30:58.410 INFO:teuthology.run_tasks:Running task clock... 2023-10-04T00:30:58.420 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-04T00:30:58.420 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-04T00:30:58.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Command line: ntpd -gq 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: ---------------------------------------------------- 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: ntp-4 is maintained by Network Time Foundation, 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: corporation. Support and training for ntp-4 are 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: available at https://www.nwtime.org/support 2023-10-04T00:30:58.480 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: ---------------------------------------------------- 2023-10-04T00:30:58.481 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: proto: precision = 0.061 usec (-24) 2023-10-04T00:30:58.481 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: basedate set to 2022-02-04 2023-10-04T00:30:58.481 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: gps base set to 2022-02-06 (week 2196) 2023-10-04T00:30:58.481 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-04T00:30:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-04T00:30:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: restrict ::: KOD does nothing without LIMITED. 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen and drop on 0 v6wildcard [::]:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen normally on 2 lo 127.0.0.1:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen normally on 3 ens1f0 172.21.15.27:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen normally on 4 lo [::1]:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:395a%4]:123 2023-10-04T00:30:58.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:30:58 ntpd[27711]: Listening on routing socket on fd #22 for interface updates 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:30:59.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:30:59 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:01.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:01.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:01.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:01.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:01.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:01.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:01 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:03.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:03.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:03.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:03.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:03.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:03.484 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:03 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:05.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:05.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:05.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:05.481 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/rawstats.20231004: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-04T00:31:05.482 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/peerstats.20231004: Permission denied 2023-10-04T00:31:05.483 INFO:teuthology.orchestra.run.smithi027.stdout: 4 Oct 00:31:05 ntpd[27711]: ntpd: time slew -0.002307 s 2023-10-04T00:31:05.483 INFO:teuthology.orchestra.run.smithi027.stdout:ntpd: time slew -0.002307s 2023-10-04T00:31:05.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-04T00:31:05.483 INFO:teuthology.orchestra.run.smithi027.stderr: 4 Oct 00:31:05 ntpd[27711]: can't open /var/log/ntpstats/loopstats.20231004: Permission denied 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: remote refid st t when poll reach delay offset jitter 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================== 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-04T00:31:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-04T00:31:05.558 INFO:teuthology.run_tasks:Running task install... 2023-10-04T00:31:05.567 DEBUG:teuthology.task.install:project ceph 2023-10-04T00:31:05.567 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-04T00:31:05.567 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-04T00:31:05.567 INFO:teuthology.task.install:Using flavor: default 2023-10-04T00:31:05.572 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-04T00:31:05.572 INFO:teuthology.task.install:extra packages: [] 2023-10-04T00:31:05.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-key list | grep Ceph 2023-10-04T00:31:05.658 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-04T00:31:05.684 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-04T00:31:05.685 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph.com (release key) 2023-10-04T00:31:05.685 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-04T00:31:05.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-04T00:31:05.827 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-04T00:31:05.828 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:31:06.117 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:31:06.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-04T00:31:06.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get update 2023-10-04T00:31:06.432 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-04T00:31:06.497 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-04T00:31:06.553 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-04T00:31:06.612 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-04T00:31:06.669 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.6 kB] 2023-10-04T00:31:06.736 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-04T00:31:07.467 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-04T00:31:07.509 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-04T00:31:07.551 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-04T00:31:07.793 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 28.6 kB in 1s (19.4 kB/s) 2023-10-04T00:31:08.690 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:31:08.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:31:08.768 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:31:08.958 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2023-10-04T00:31:08.959 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2023-10-04T00:31:09.131 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2023-10-04T00:31:09.131 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-04T00:31:09.131 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-04T00:31:09.132 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-04T00:31:09.132 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-04T00:31:09.132 INFO:teuthology.orchestra.run.smithi027.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-04T00:31:09.132 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-04T00:31:09.133 INFO:teuthology.orchestra.run.smithi027.stdout:The following additional packages will be installed: 2023-10-04T00:31:09.133 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-04T00:31:09.133 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-04T00:31:09.133 INFO:teuthology.orchestra.run.smithi027.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-04T00:31:09.133 INFO:teuthology.orchestra.run.smithi027.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-04T00:31:09.134 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-04T00:31:09.134 INFO:teuthology.orchestra.run.smithi027.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-04T00:31:09.135 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-04T00:31:09.136 INFO:teuthology.orchestra.run.smithi027.stdout:Suggested packages: 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: python3-watchdog 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout:Recommended packages: 2023-10-04T00:31:09.137 INFO:teuthology.orchestra.run.smithi027.stdout: btrfs-tools 2023-10-04T00:31:09.238 INFO:teuthology.orchestra.run.smithi027.stdout:The following NEW packages will be installed: 2023-10-04T00:31:09.238 INFO:teuthology.orchestra.run.smithi027.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-04T00:31:09.238 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-04T00:31:09.239 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-04T00:31:09.239 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-04T00:31:09.240 INFO:teuthology.orchestra.run.smithi027.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-04T00:31:09.241 INFO:teuthology.orchestra.run.smithi027.stdout: socat xmlstarlet 2023-10-04T00:31:09.301 INFO:teuthology.orchestra.run.smithi027.stdout:0 upgraded, 109 newly installed, 0 to remove and 254 not upgraded. 2023-10-04T00:31:09.301 INFO:teuthology.orchestra.run.smithi027.stdout:Need to get 183 MB of archives. 2023-10-04T00:31:09.301 INFO:teuthology.orchestra.run.smithi027.stdout:After this operation, 823 MB of additional disk space will be used. 2023-10-04T00:31:09.302 INFO:teuthology.orchestra.run.smithi027.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-04T00:31:09.359 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-04T00:31:09.394 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-04T00:31:09.397 INFO:teuthology.orchestra.run.smithi027.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-04T00:31:09.412 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-04T00:31:09.463 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-04T00:31:09.467 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-04T00:31:09.479 INFO:teuthology.orchestra.run.smithi027.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-04T00:31:09.483 INFO:teuthology.orchestra.run.smithi027.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-04T00:31:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-04T00:31:09.484 INFO:teuthology.orchestra.run.smithi027.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-04T00:31:09.493 INFO:teuthology.orchestra.run.smithi027.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-04T00:31:09.494 INFO:teuthology.orchestra.run.smithi027.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-04T00:31:09.494 INFO:teuthology.orchestra.run.smithi027.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-04T00:31:09.495 INFO:teuthology.orchestra.run.smithi027.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-04T00:31:09.501 INFO:teuthology.orchestra.run.smithi027.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-04T00:31:09.501 INFO:teuthology.orchestra.run.smithi027.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-04T00:31:09.518 INFO:teuthology.orchestra.run.smithi027.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-04T00:31:09.519 INFO:teuthology.orchestra.run.smithi027.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-04T00:31:09.519 INFO:teuthology.orchestra.run.smithi027.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-04T00:31:09.520 INFO:teuthology.orchestra.run.smithi027.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-04T00:31:09.522 INFO:teuthology.orchestra.run.smithi027.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-04T00:31:09.523 INFO:teuthology.orchestra.run.smithi027.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-04T00:31:09.523 INFO:teuthology.orchestra.run.smithi027.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-04T00:31:09.524 INFO:teuthology.orchestra.run.smithi027.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-04T00:31:09.535 INFO:teuthology.orchestra.run.smithi027.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-04T00:31:09.536 INFO:teuthology.orchestra.run.smithi027.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-04T00:31:09.552 INFO:teuthology.orchestra.run.smithi027.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-04T00:31:09.553 INFO:teuthology.orchestra.run.smithi027.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-04T00:31:09.554 INFO:teuthology.orchestra.run.smithi027.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-04T00:31:09.554 INFO:teuthology.orchestra.run.smithi027.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-04T00:31:09.560 INFO:teuthology.orchestra.run.smithi027.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-04T00:31:09.560 INFO:teuthology.orchestra.run.smithi027.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-04T00:31:09.561 INFO:teuthology.orchestra.run.smithi027.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-04T00:31:09.561 INFO:teuthology.orchestra.run.smithi027.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-04T00:31:09.569 INFO:teuthology.orchestra.run.smithi027.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-04T00:31:09.571 INFO:teuthology.orchestra.run.smithi027.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-04T00:31:09.580 INFO:teuthology.orchestra.run.smithi027.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-04T00:31:09.581 INFO:teuthology.orchestra.run.smithi027.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-04T00:31:09.582 INFO:teuthology.orchestra.run.smithi027.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-04T00:31:09.585 INFO:teuthology.orchestra.run.smithi027.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-04T00:31:09.585 INFO:teuthology.orchestra.run.smithi027.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-04T00:31:09.588 INFO:teuthology.orchestra.run.smithi027.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-04T00:31:09.618 INFO:teuthology.orchestra.run.smithi027.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6336-ga7de91bf-1jammy [3,437 kB] 2023-10-04T00:31:09.642 INFO:teuthology.orchestra.run.smithi027.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-04T00:31:09.643 INFO:teuthology.orchestra.run.smithi027.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-04T00:31:09.644 INFO:teuthology.orchestra.run.smithi027.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-04T00:31:09.645 INFO:teuthology.orchestra.run.smithi027.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-04T00:31:09.645 INFO:teuthology.orchestra.run.smithi027.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-04T00:31:09.646 INFO:teuthology.orchestra.run.smithi027.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-04T00:31:09.663 INFO:teuthology.orchestra.run.smithi027.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-04T00:31:09.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-04T00:31:09.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-04T00:31:09.680 INFO:teuthology.orchestra.run.smithi027.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-04T00:31:09.680 INFO:teuthology.orchestra.run.smithi027.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-04T00:31:09.680 INFO:teuthology.orchestra.run.smithi027.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-04T00:31:09.681 INFO:teuthology.orchestra.run.smithi027.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-04T00:31:09.681 INFO:teuthology.orchestra.run.smithi027.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-04T00:31:09.682 INFO:teuthology.orchestra.run.smithi027.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-04T00:31:09.694 INFO:teuthology.orchestra.run.smithi027.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-04T00:31:09.695 INFO:teuthology.orchestra.run.smithi027.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-04T00:31:09.698 INFO:teuthology.orchestra.run.smithi027.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-04T00:31:09.711 INFO:teuthology.orchestra.run.smithi027.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-04T00:31:09.713 INFO:teuthology.orchestra.run.smithi027.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-04T00:31:09.713 INFO:teuthology.orchestra.run.smithi027.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-04T00:31:09.716 INFO:teuthology.orchestra.run.smithi027.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-04T00:31:09.718 INFO:teuthology.orchestra.run.smithi027.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-04T00:31:09.723 INFO:teuthology.orchestra.run.smithi027.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-04T00:31:09.725 INFO:teuthology.orchestra.run.smithi027.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-04T00:31:09.728 INFO:teuthology.orchestra.run.smithi027.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-04T00:31:09.729 INFO:teuthology.orchestra.run.smithi027.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-04T00:31:09.729 INFO:teuthology.orchestra.run.smithi027.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-04T00:31:09.745 INFO:teuthology.orchestra.run.smithi027.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-04T00:31:09.746 INFO:teuthology.orchestra.run.smithi027.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-04T00:31:09.752 INFO:teuthology.orchestra.run.smithi027.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-04T00:31:09.752 INFO:teuthology.orchestra.run.smithi027.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-04T00:31:09.753 INFO:teuthology.orchestra.run.smithi027.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-04T00:31:09.754 INFO:teuthology.orchestra.run.smithi027.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-04T00:31:09.755 INFO:teuthology.orchestra.run.smithi027.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-04T00:31:10.107 INFO:teuthology.orchestra.run.smithi027.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6336-ga7de91bf-1jammy [3,121 kB] 2023-10-04T00:31:10.285 INFO:teuthology.orchestra.run.smithi027.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6336-ga7de91bf-1jammy [676 kB] 2023-10-04T00:31:10.322 INFO:teuthology.orchestra.run.smithi027.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6336-ga7de91bf-1jammy [345 kB] 2023-10-04T00:31:10.341 INFO:teuthology.orchestra.run.smithi027.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6336-ga7de91bf-1jammy [23.0 kB] 2023-10-04T00:31:10.342 INFO:teuthology.orchestra.run.smithi027.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6336-ga7de91bf-1jammy [170 kB] 2023-10-04T00:31:10.358 INFO:teuthology.orchestra.run.smithi027.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6336-ga7de91bf-1jammy [52.4 kB] 2023-10-04T00:31:10.361 INFO:teuthology.orchestra.run.smithi027.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6336-ga7de91bf-1jammy [318 kB] 2023-10-04T00:31:10.370 INFO:teuthology.orchestra.run.smithi027.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6336-ga7de91bf-1jammy [6,201 kB] 2023-10-04T00:31:10.667 INFO:teuthology.orchestra.run.smithi027.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6336-ga7de91bf-1jammy [102 kB] 2023-10-04T00:31:10.669 INFO:teuthology.orchestra.run.smithi027.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6336-ga7de91bf-1jammy [428 kB] 2023-10-04T00:31:10.688 INFO:teuthology.orchestra.run.smithi027.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6336-ga7de91bf-1jammy [22.5 MB] 2023-10-04T00:31:11.547 INFO:teuthology.orchestra.run.smithi027.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6336-ga7de91bf-1jammy [6,840 kB] 2023-10-04T00:31:11.707 INFO:teuthology.orchestra.run.smithi027.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6336-ga7de91bf-1jammy [228 kB] 2023-10-04T00:31:11.713 INFO:teuthology.orchestra.run.smithi027.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6336-ga7de91bf-1jammy [117 kB] 2023-10-04T00:31:11.718 INFO:teuthology.orchestra.run.smithi027.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6336-ga7de91bf-1jammy [1,487 kB] 2023-10-04T00:31:11.766 INFO:teuthology.orchestra.run.smithi027.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6336-ga7de91bf-1jammy [9,851 kB] 2023-10-04T00:31:12.093 INFO:teuthology.orchestra.run.smithi027.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6336-ga7de91bf-1jammy [31.7 MB] 2023-10-04T00:31:13.236 INFO:teuthology.orchestra.run.smithi027.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6336-ga7de91bf-1jammy [3,596 B] 2023-10-04T00:31:13.237 INFO:teuthology.orchestra.run.smithi027.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [827 kB] 2023-10-04T00:31:13.253 INFO:teuthology.orchestra.run.smithi027.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6336-ga7de91bf-1jammy [2,135 kB] 2023-10-04T00:31:13.291 INFO:teuthology.orchestra.run.smithi027.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6336-ga7de91bf-1jammy [251 kB] 2023-10-04T00:31:13.295 INFO:teuthology.orchestra.run.smithi027.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6336-ga7de91bf-1jammy [128 kB] 2023-10-04T00:31:13.296 INFO:teuthology.orchestra.run.smithi027.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6336-ga7de91bf-1jammy [1,883 kB] 2023-10-04T00:31:13.347 INFO:teuthology.orchestra.run.smithi027.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6336-ga7de91bf-1jammy [8,615 kB] 2023-10-04T00:31:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6336-ga7de91bf-1jammy [3,722 B] 2023-10-04T00:31:13.635 INFO:teuthology.orchestra.run.smithi027.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6336-ga7de91bf-1jammy [47.7 MB] 2023-10-04T00:31:15.502 INFO:teuthology.orchestra.run.smithi027.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6336-ga7de91bf-1jammy [234 kB] 2023-10-04T00:31:15.575 INFO:teuthology.orchestra.run.smithi027.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6336-ga7de91bf-1jammy [28.3 kB] 2023-10-04T00:31:15.576 INFO:teuthology.orchestra.run.smithi027.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6336-ga7de91bf-1jammy [12.3 MB] 2023-10-04T00:31:15.902 INFO:teuthology.orchestra.run.smithi027.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6336-ga7de91bf-1jammy [80.9 kB] 2023-10-04T00:31:16.475 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 183 MB in 7s (27.6 MB/s) 2023-10-04T00:31:16.613 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-04T00:31:16.711 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2023-10-04T00:31:16.716 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-04T00:31:16.803 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-04T00:31:17.122 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-04T00:31:17.149 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-04T00:31:17.189 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-04T00:31:17.534 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-04T00:31:17.561 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-04T00:31:17.601 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-04T00:31:17.912 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-04T00:31:17.939 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-04T00:31:17.978 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-04T00:31:18.457 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-04T00:31:18.484 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-04T00:31:18.532 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:31:18.918 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-04T00:31:18.945 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-04T00:31:18.984 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:31:19.330 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-04T00:31:19.356 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-04T00:31:19.397 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:31:19.766 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-04T00:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-04T00:31:19.832 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-04T00:31:20.160 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librados2. 2023-10-04T00:31:20.187 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../008-librados2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:20.227 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:20.638 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librbd1. 2023-10-04T00:31:20.665 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../009-librbd1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:20.696 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:21.133 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libcephfs2. 2023-10-04T00:31:21.159 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:21.199 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:21.493 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rados. 2023-10-04T00:31:21.509 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:21.551 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:22.121 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-04T00:31:22.148 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:22.179 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:22.415 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cephfs. 2023-10-04T00:31:22.442 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:22.482 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:22.793 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-04T00:31:22.820 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:22.859 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:23.196 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-04T00:31:23.223 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-04T00:31:23.262 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-04T00:31:23.558 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-prettytable. 2023-10-04T00:31:23.573 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-04T00:31:23.607 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-04T00:31:23.843 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rbd. 2023-10-04T00:31:23.869 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:23.909 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:24.279 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-04T00:31:24.306 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-04T00:31:24.345 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-04T00:31:24.682 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librgw2. 2023-10-04T00:31:24.708 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../019-librgw2_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:24.748 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:25.250 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rgw. 2023-10-04T00:31:25.267 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:25.300 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:25.637 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-04T00:31:25.663 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-04T00:31:25.703 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-04T00:31:26.048 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libradosstriper1. 2023-10-04T00:31:26.075 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:26.114 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:26.468 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-common. 2023-10-04T00:31:26.494 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:26.526 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:28.074 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-base. 2023-10-04T00:31:28.100 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:28.149 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:28.869 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-04T00:31:28.896 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-04T00:31:28.936 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-04T00:31:29.239 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cheroot. 2023-10-04T00:31:29.265 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-04T00:31:29.305 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-04T00:31:29.650 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-04T00:31:29.676 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-04T00:31:29.741 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-04T00:31:30.010 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-04T00:31:30.026 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-04T00:31:30.069 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-04T00:31:30.372 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-04T00:31:30.399 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-04T00:31:30.438 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-04T00:31:30.741 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-tempora. 2023-10-04T00:31:30.768 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-04T00:31:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-04T00:31:31.085 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-portend. 2023-10-04T00:31:31.101 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-04T00:31:31.135 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-04T00:31:31.414 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-04T00:31:31.440 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-04T00:31:31.480 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-04T00:31:31.783 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-04T00:31:31.809 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-04T00:31:31.849 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-04T00:31:32.211 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-natsort. 2023-10-04T00:31:32.238 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-04T00:31:32.277 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-04T00:31:32.556 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-logutils. 2023-10-04T00:31:32.572 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-04T00:31:32.605 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-04T00:31:32.908 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-04T00:31:32.935 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-04T00:31:32.975 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-04T00:31:33.278 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-mako. 2023-10-04T00:31:33.304 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-04T00:31:33.344 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-04T00:31:33.698 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-04T00:31:33.724 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-04T00:31:33.764 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-04T00:31:34.009 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-04T00:31:34.035 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-04T00:31:34.067 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-04T00:31:34.361 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-webob. 2023-10-04T00:31:34.388 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-04T00:31:34.428 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-04T00:31:34.781 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-waitress. 2023-10-04T00:31:34.808 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-04T00:31:34.851 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-04T00:31:35.159 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-tempita. 2023-10-04T00:31:35.186 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-04T00:31:35.225 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-04T00:31:35.528 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-paste. 2023-10-04T00:31:35.555 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-04T00:31:35.594 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-04T00:31:35.914 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-04T00:31:35.930 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-04T00:31:35.973 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-04T00:31:36.275 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-04T00:31:36.302 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-04T00:31:36.348 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-04T00:31:36.628 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-webtest. 2023-10-04T00:31:36.652 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-04T00:31:36.686 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-04T00:31:36.998 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pecan. 2023-10-04T00:31:37.025 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-04T00:31:37.064 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-04T00:31:37.444 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-certifi. 2023-10-04T00:31:37.471 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-04T00:31:37.510 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-04T00:31:38.056 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-idna. 2023-10-04T00:31:38.082 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-04T00:31:38.122 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-04T00:31:38.433 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-urllib3. 2023-10-04T00:31:38.460 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-04T00:31:38.500 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-04T00:31:38.845 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-requests. 2023-10-04T00:31:38.871 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-04T00:31:38.911 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-04T00:31:39.206 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-04T00:31:39.233 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-04T00:31:39.272 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-04T00:31:39.567 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-04T00:31:39.594 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:39.633 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:39.996 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-04T00:31:40.023 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:40.054 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:40.358 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-openssl. 2023-10-04T00:31:40.385 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-04T00:31:40.424 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-04T00:31:40.777 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr. 2023-10-04T00:31:40.804 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:40.843 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:41.272 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mon. 2023-10-04T00:31:41.299 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:41.338 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:42.892 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-osd. 2023-10-04T00:31:42.919 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:42.958 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:44.870 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph. 2023-10-04T00:31:44.897 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../059-ceph_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:44.936 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:45.273 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-fuse. 2023-10-04T00:31:45.300 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:45.331 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:45.684 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mds. 2023-10-04T00:31:45.711 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:45.751 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:46.137 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package cephadm. 2023-10-04T00:31:46.164 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../062-cephadm_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:46.203 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:46.498 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-04T00:31:46.525 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-04T00:31:46.556 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-04T00:31:46.876 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-04T00:31:46.892 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-04T00:31:46.934 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-04T00:31:47.989 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-babel. 2023-10-04T00:31:48.017 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-04T00:31:48.056 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-04T00:31:48.609 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jinja2. 2023-10-04T00:31:48.636 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-04T00:31:48.675 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-04T00:31:49.045 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-04T00:31:49.072 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:49.112 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:49.466 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-04T00:31:49.493 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-04T00:31:49.532 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-04T00:31:49.793 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-04T00:31:49.809 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-04T00:31:49.843 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-04T00:31:50.104 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-routes. 2023-10-04T00:31:50.120 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-04T00:31:50.154 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-04T00:31:50.441 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-04T00:31:50.468 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:50.524 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:50.961 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-04T00:31:50.988 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-04T00:31:51.027 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-04T00:31:51.556 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-joblib. 2023-10-04T00:31:51.583 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-04T00:31:51.622 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-04T00:31:51.984 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-04T00:31:52.012 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-04T00:31:52.050 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-04T00:31:52.337 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-sklearn. 2023-10-04T00:31:52.364 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-04T00:31:52.403 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-04T00:31:53.112 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-04T00:31:53.140 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:53.178 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:54.182 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cachetools. 2023-10-04T00:31:54.209 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-04T00:31:54.248 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-04T00:31:54.551 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-04T00:31:54.578 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-04T00:31:54.617 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-04T00:31:54.904 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-04T00:31:54.920 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-04T00:31:54.953 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-04T00:31:55.259 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rsa. 2023-10-04T00:31:55.286 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-04T00:31:55.325 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-04T00:31:55.646 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-google-auth. 2023-10-04T00:31:55.673 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-04T00:31:55.712 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-04T00:31:56.032 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-04T00:31:56.059 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-04T00:31:56.098 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-04T00:31:56.478 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-websocket. 2023-10-04T00:31:56.506 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-04T00:31:56.536 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-04T00:31:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-04T00:31:56.909 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-04T00:31:56.947 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-04T00:31:57.662 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-04T00:31:57.678 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:31:57.720 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:31:58.023 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-04T00:31:58.051 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-04T00:31:58.089 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-04T00:31:58.401 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-04T00:31:58.429 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-04T00:31:58.467 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-04T00:31:58.720 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package jq. 2023-10-04T00:31:58.748 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-04T00:31:58.787 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-04T00:31:59.014 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package socat. 2023-10-04T00:31:59.030 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-04T00:31:59.064 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-04T00:31:59.343 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package xmlstarlet. 2023-10-04T00:31:59.359 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-04T00:31:59.401 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-04T00:31:59.789 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-test. 2023-10-04T00:31:59.816 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:31:59.855 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:02.100 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-volume. 2023-10-04T00:32:02.117 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6336-ga7de91bf-1jammy_all.deb ... 2023-10-04T00:32:02.159 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:02.530 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-04T00:32:02.546 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:32:02.588 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:02.958 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package nvme-cli. 2023-10-04T00:32:02.986 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-04T00:32:03.133 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-04T00:32:03.497 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-04T00:32:03.513 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-04T00:32:03.555 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-04T00:32:04.309 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-04T00:32:04.337 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-04T00:32:04.375 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-04T00:32:04.644 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pastescript. 2023-10-04T00:32:04.672 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-04T00:32:04.702 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-04T00:32:05.014 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pluggy. 2023-10-04T00:32:05.042 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-04T00:32:05.081 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-04T00:32:05.392 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-psutil. 2023-10-04T00:32:05.420 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-04T00:32:05.459 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-04T00:32:05.795 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-py. 2023-10-04T00:32:05.823 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-04T00:32:05.862 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-04T00:32:06.190 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pygments. 2023-10-04T00:32:06.218 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-04T00:32:06.256 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-04T00:32:06.744 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-04T00:32:06.772 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-04T00:32:06.811 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-04T00:32:07.105 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-toml. 2023-10-04T00:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-04T00:32:07.171 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-04T00:32:07.433 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pytest. 2023-10-04T00:32:07.449 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-04T00:32:07.482 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-04T00:32:07.820 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-simplejson. 2023-10-04T00:32:07.848 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-04T00:32:07.886 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-04T00:32:08.189 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-04T00:32:08.217 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-04T00:32:08.264 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-04T00:32:09.035 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package radosgw. 2023-10-04T00:32:09.063 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../107-radosgw_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:32:09.101 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:09.871 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package rbd-fuse. 2023-10-04T00:32:09.899 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6336-ga7de91bf-1jammy_amd64.deb ... 2023-10-04T00:32:09.929 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:10.217 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-04T00:32:10.484 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-04T00:32:10.610 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-04T00:32:10.911 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-04T00:32:11.275 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-04T00:32:11.733 INFO:teuthology.orchestra.run.smithi027.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-04T00:32:11.828 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:12.380 INFO:teuthology.orchestra.run.smithi027.stdout:Adding system user cephadm....done 2023-10-04T00:32:12.526 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-04T00:32:12.796 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-04T00:32:13.046 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-04T00:32:13.188 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-04T00:32:13.448 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-04T00:32:13.724 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-04T00:32:13.850 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-04T00:32:14.152 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-04T00:32:14.521 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-04T00:32:14.780 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-04T00:32:15.039 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-04T00:32:15.299 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-04T00:32:15.558 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-04T00:32:15.818 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-ceph-argparse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:16.219 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-04T00:32:16.353 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-04T00:32:16.613 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-04T00:32:16.739 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-04T00:32:17.106 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-04T00:32:17.232 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-04T00:32:17.517 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-04T00:32:17.760 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-04T00:32:18.061 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-04T00:32:18.730 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-04T00:32:18.989 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-04T00:32:19.115 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-04T00:32:19.241 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-04T00:32:19.642 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-04T00:32:20.019 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-04T00:32:20.261 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-04T00:32:20.646 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-04T00:32:20.939 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-04T00:32:21.291 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-04T00:32:21.550 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-04T00:32:21.651 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:21.936 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-idna (3.3-1) ... 2023-10-04T00:32:22.262 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-04T00:32:23.484 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:32:23.602 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-04T00:32:23.862 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-04T00:32:23.987 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-04T00:32:24.113 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-04T00:32:24.415 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-04T00:32:24.674 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-04T00:32:24.933 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:32:25.035 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-04T00:32:25.144 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-04T00:32:25.437 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-04T00:32:25.705 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-04T00:32:25.973 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-04T00:32:26.232 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-04T00:32:26.491 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-04T00:32:26.759 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-04T00:32:26.869 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-04T00:32:27.145 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-04T00:32:27.263 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-04T00:32:27.581 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-04T00:32:27.890 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-04T00:32:28.149 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-04T00:32:28.542 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-04T00:32:28.827 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-04T00:32:28.928 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-04T00:32:29.321 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-04T00:32:29.589 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-04T00:32:29.848 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-04T00:32:29.991 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-04T00:32:30.309 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-04T00:32:30.610 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-04T00:32:30.870 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-04T00:32:30.979 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-04T00:32:31.255 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-04T00:32:31.539 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-04T00:32:31.824 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-04T00:32:32.217 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-04T00:32:32.502 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-04T00:32:32.846 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-04T00:32:32.972 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librados2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:33.098 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librgw2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:33.224 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:33.350 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libcephfs2 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:33.476 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libradosstriper1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:33.671 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-04T00:32:33.955 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librbd1 (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:34.082 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-modules-core (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:34.208 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-04T00:32:34.408 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-04T00:32:34.909 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libcephfs-dev (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:35.019 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rados (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:35.145 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-04T00:32:36.449 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rbd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:36.550 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up rbd-fuse (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:36.652 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:36.769 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cephfs (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:36.895 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-common (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:37.438 INFO:teuthology.orchestra.run.smithi027.stdout:Adding group ceph....done 2023-10-04T00:32:37.790 INFO:teuthology.orchestra.run.smithi027.stdout:Adding system user ceph....done 2023-10-04T00:32:37.966 INFO:teuthology.orchestra.run.smithi027.stdout:Setting system user ceph properties....done 2023-10-04T00:32:37.980 INFO:teuthology.orchestra.run.smithi027.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-04T00:32:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-04T00:32:38.489 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-04T00:32:39.013 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-test (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:39.164 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up radosgw (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:39.438 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-04T00:32:39.439 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-04T00:32:40.458 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-base (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:40.809 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-04T00:32:41.318 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mds (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:41.535 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-04T00:32:41.535 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-04T00:32:42.028 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:42.242 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-04T00:32:42.242 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-04T00:32:42.754 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-osd (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:43.075 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-04T00:32:43.075 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-04T00:32:43.605 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:43.882 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mon (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:44.083 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-04T00:32:44.083 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-04T00:32:44.561 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-cephadm (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:44.712 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:44.838 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-dashboard (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:44.989 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-volume (18.0.0-6336-ga7de91bf-1jammy) ... 2023-10-04T00:32:45.116 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-04T00:32:46.221 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-04T00:32:46.681 INFO:teuthology.orchestra.run.smithi027.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-04T00:32:46.684 DEBUG:teuthology.parallel:result is None 2023-10-04T00:32:46.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-04T00:32:46.828 DEBUG:teuthology.orchestra.run.smithi027:> dpkg-query -W -f '${Version}' ceph 2023-10-04T00:32:46.847 INFO:teuthology.orchestra.run.smithi027.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:32:46.848 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:32:46.848 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-04T00:32:46.848 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-04T00:32:46.848 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:46.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-04T00:32:46.902 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-04T00:32:46.903 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:46.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2023-10-04T00:32:46.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-04T00:32:47.018 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-04T00:32:47.018 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:47.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-04T00:32:47.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-04T00:32:47.129 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-04T00:32:47.130 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:47.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2023-10-04T00:32:47.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-04T00:32:47.238 INFO:teuthology.run_tasks:Running task ceph... 2023-10-04T00:32:47.296 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-04T00:32:47.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2023-10-04T00:32:47.308 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-04T00:32:47.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-04T00:32:47.362 INFO:tasks.ceph:Creating extra log directories... 2023-10-04T00:32:47.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-04T00:32:47.426 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-04T00:32:47.426 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-04T00:32:47.427 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079', 'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7410079', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ajarr-check-addr-non-zero', 'sha1': '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 7470, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-ajarr-check-addr-non-zero', 'suite_path': '/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ajarr/ceph', 'suite_sha1': '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0ecNHL7KfIGkixkKWDByilxad+DFHOUw5+x9jVxBF8gvTOZ1tVfoPG8cDk7ttC44bcMPg9G/SHv/LmT/cUSy8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-04_00:09:14', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-04T00:32:47.427 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-04T00:32:47.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2023-10-04T00:32:47.523 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:47.523 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2023-10-04T00:32:47.571 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-04T00:32:47.571 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2023-10-04T00:32:47.619 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-04T00:32:47.619 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-04T00:32:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 862 Links: 1 2023-10-04T00:32:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-04T00:32:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-04 00:27:37.915459918 +0000 2023-10-04T00:32:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-04 00:27:37.563463061 +0000 2023-10-04T00:32:47.620 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-04 00:27:37.563463061 +0000 2023-10-04T00:32:47.621 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-04T00:32:47.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-04T00:32:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-04T00:32:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-04T00:32:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00038745 s, 1.3 MB/s 2023-10-04T00:32:47.675 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-04T00:32:47.724 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2023-10-04T00:32:47.775 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-04T00:32:47.775 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-04T00:32:47.775 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 884 Links: 1 2023-10-04T00:32:47.775 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-04T00:32:47.775 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-04 00:27:38.479454881 +0000 2023-10-04T00:32:47.776 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-04 00:27:38.147457846 +0000 2023-10-04T00:32:47.776 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-04 00:27:38.147457846 +0000 2023-10-04T00:32:47.776 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-04T00:32:47.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-04T00:32:47.829 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-04T00:32:47.829 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-04T00:32:47.829 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000437386 s, 1.2 MB/s 2023-10-04T00:32:47.831 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-04T00:32:47.880 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2023-10-04T00:32:47.931 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-04T00:32:47.931 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-04T00:32:47.931 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-10-04T00:32:47.931 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-04T00:32:47.932 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-04 00:27:39.051449770 +0000 2023-10-04T00:32:47.932 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-04 00:27:38.715452772 +0000 2023-10-04T00:32:47.932 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-04 00:27:38.715452772 +0000 2023-10-04T00:32:47.932 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-04T00:32:47.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-04T00:32:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-04T00:32:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-04T00:32:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000427317 s, 1.2 MB/s 2023-10-04T00:32:47.991 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-04T00:32:48.040 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2023-10-04T00:32:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-04T00:32:48.087 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-04T00:32:48.087 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 918 Links: 1 2023-10-04T00:32:48.087 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-04T00:32:48.088 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-04 00:27:39.623444662 +0000 2023-10-04T00:32:48.088 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-04 00:27:39.271447805 +0000 2023-10-04T00:32:48.088 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-04 00:27:39.271447805 +0000 2023-10-04T00:32:48.088 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-04T00:32:48.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-04T00:32:48.141 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-04T00:32:48.141 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-04T00:32:48.142 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000347005 s, 1.5 MB/s 2023-10-04T00:32:48.143 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-04T00:32:48.192 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-04T00:32:48.192 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-04T00:32:48.192 INFO:tasks.ceph:Generating config... 2023-10-04T00:32:48.193 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-04T00:32:48.193 INFO:tasks.ceph:[client] rbd default features = 61 2023-10-04T00:32:48.193 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-04T00:32:48.193 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-04T00:32:48.194 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-04T00:32:48.194 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-04T00:32:48.194 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-04T00:32:48.194 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-04T00:32:48.194 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-04T00:32:48.194 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-04T00:32:48.195 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-04T00:32:48.196 INFO:tasks.ceph:Setting up mon.a... 2023-10-04T00:32:48.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-04T00:32:48.270 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2023-10-04T00:32:48.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-04T00:32:48.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-04T00:32:48.409 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27')] 2023-10-04T00:32:48.410 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2023-10-04T00:32:48.411 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:48.411 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-04T00:32:48.455 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:48.534 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 250ac99b-0ea2-47d3-80de-42417fc97a77 2023-10-04T00:32:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = pacific 2023-10-04T00:32:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2023-10-04T00:32:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 250ac99b-0ea2-47d3-80de-42417fc97a77 2023-10-04T00:32:48.535 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2023-10-04T00:32:48.534746+0000 2023-10-04T00:32:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:created 2023-10-04T00:32:48.534746+0000 2023-10-04T00:32:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 16 (pacific) 2023-10-04T00:32:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2023-10-04T00:32:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2023-10-04T00:32:48.536 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-04T00:32:48.537 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-04T00:32:48.583 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 250ac99b-0ea2-47d3-80de-42417fc97a77... 2023-10-04T00:32:48.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-04T00:32:48.657 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2023-10-04T00:32:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2023-10-04T00:32:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-04T00:32:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2023-10-04T00:32:48.657 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2023-10-04T00:32:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2023-10-04T00:32:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.658 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.659 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.660 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2023-10-04T00:32:48.661 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2023-10-04T00:32:48.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.663 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = 172.21.15.27 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: mon client directed command retry = 5 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on pool no app = False 2023-10-04T00:32:48.664 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject socket failures = 5000 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 250ac99b-0ea2-47d3-80de-42417fc97a77 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.665 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2023-10-04T00:32:48.666 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2023-10-04T00:32:48.667 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2023-10-04T00:32:48.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2023-10-04T00:32:48.669 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2023-10-04T00:32:48.670 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2023-10-04T00:32:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: osd shutdown pgref assert = True 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-04T00:32:48.672 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2023-10-04T00:32:48.673 INFO:teuthology.orchestra.run.smithi027.stdout: debug rbd = 20 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2023-10-04T00:32:48.674 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2023-10-04T00:32:48.675 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.676 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2023-10-04T00:32:48.677 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2023-10-04T00:32:48.678 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2023-10-04T00:32:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-04T00:32:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-04T00:32:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: rbd default data pool = datapool 2023-10-04T00:32:48.679 INFO:teuthology.orchestra.run.smithi027.stdout: rbd default features = 61 2023-10-04T00:32:48.679 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2023-10-04T00:32:48.680 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-04T00:32:48.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-04T00:32:48.774 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-04T00:32:48.775 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:48.775 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-04T00:32:48.780 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:48.780 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-04T00:32:48.826 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:32:48.826 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:48.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-04T00:32:48.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-04T00:32:48.891 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:32:48.891 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:48.943 INFO:tasks.ceph:Setting up mon nodes... 2023-10-04T00:32:48.943 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-04T00:32:48.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-04T00:32:49.029 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-04T00:32:49.032 INFO:tasks.ceph:Setting up mds nodes... 2023-10-04T00:32:49.032 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-04T00:32:49.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-04T00:32:49.108 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-04T00:32:49.120 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-04T00:32:49.120 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-04T00:32:49.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-04T00:32:49.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-04T00:32:49.174 INFO:tasks.ceph:role: osd.0 2023-10-04T00:32:49.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:32:49.174 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-04T00:32:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-04T00:32:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-04T00:32:49.690 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-04T00:32:49.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-04T00:32:49.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-04T00:32:49.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-04T00:32:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T00:32:49.792 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2023-10-04T00:32:49.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-04T00:32:49.844 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-04T00:32:49.844 INFO:tasks.ceph:role: osd.1 2023-10-04T00:32:49.844 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:32:49.844 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-04T00:32:49.907 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-04T00:32:49.908 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-04T00:32:49.908 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-04T00:32:49.908 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-04T00:32:50.372 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-04T00:32:50.374 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-04T00:32:50.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-04T00:32:50.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-04T00:32:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T00:32:50.538 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2023-10-04T00:32:50.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-04T00:32:50.596 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-04T00:32:50.596 INFO:tasks.ceph:role: osd.2 2023-10-04T00:32:50.596 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:32:50.596 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-04T00:32:50.658 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-04T00:32:50.658 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-04T00:32:50.658 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-04T00:32:50.659 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-04T00:32:50.660 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-04T00:32:51.133 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-04T00:32:51.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-04T00:32:51.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-04T00:32:51.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-04T00:32:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T00:32:51.273 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2023-10-04T00:32:51.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:51.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-04T00:32:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-04T00:32:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-04T00:32:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-04T00:32:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-04T00:32:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:51.398+0000 7f52fe959640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-04T00:32:54.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-04T00:32:54.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-04T00:32:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-04T00:32:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-04T00:32:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-04T00:32:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-04T00:32:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:54.530+0000 7f902e170640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-04T00:32:57.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-04T00:32:57.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:32:57.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.698+0000 7f61b235f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-04T00:32:57.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.698+0000 7f61b235f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-04T00:32:57.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.698+0000 7f61b235f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-04T00:32:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.698+0000 7f61b235f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-04T00:32:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.698+0000 7f61b235f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-04T00:32:57.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-04T00:32:57.702+0000 7f61b235f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-04T00:33:00.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-04T00:33:00.763 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-04T00:33:00.763 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:00.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-04T00:33:00.818 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:00.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-04T00:33:00.874 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:00.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-04T00:33:00.930 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:00.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-04T00:33:00.985 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:00.985 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-04T00:33:01.034 INFO:tasks.ceph:Adding keys to all mons... 2023-10-04T00:33:01.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCxshxl6Y6/ARAAE3Qi0ZuNjkdttdMUigH4vg== 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCzshxlF/3ZFxAAldfwXi9pWBWwPz+HJ699sA== 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC2shxl6Ui+HxAA0QKVLzMmeGJKRRfr9jPMPg== 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC5shxlSFjIKRAAmYYx5QFbeeA4ZEJLgQLBKg== 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2023-10-04T00:33:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCxshxlcjl6BhAABVW17ZxCcM4/9TrioniwOg== 2023-10-04T00:33:01.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-04T00:33:01.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-04T00:33:01.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-04T00:33:01.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-04T00:33:01.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-04T00:33:01.478 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-04T00:33:01.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-04T00:33:01.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-04T00:33:01.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-04T00:33:01.598 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-04T00:33:01.645 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-04T00:33:01.645 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-04T00:33:01.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-04T00:33:01.687 INFO:tasks.ceph.mon.a:Started 2023-10-04T00:33:01.687 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-04T00:33:01.688 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-04T00:33:01.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-04T00:33:01.690 INFO:tasks.ceph.mgr.x:Started 2023-10-04T00:33:01.690 INFO:tasks.ceph:Setting crush tunables to default 2023-10-04T00:33:01.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2023-10-04T00:33:01.955 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2023-10-04T00:33:01.974 INFO:tasks.ceph:check_enable_crimson: False 2023-10-04T00:33:01.974 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-04T00:33:01.974 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:01.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-04T00:33:01.986 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:01.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-04T00:33:02.045 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:02.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-04T00:33:02.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new d1a54a86-80fd-4af9-ad9b-9544a78b082c 0 2023-10-04T00:33:02.149 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:02.146+0000 7ff1e591f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-04T00:33:02.348 INFO:teuthology.orchestra.run.smithi027.stdout:0 2023-10-04T00:33:02.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new d4cb688e-0dce-4f66-a7f8-ab84f8910e2a 1 2023-10-04T00:33:02.612 INFO:teuthology.orchestra.run.smithi027.stdout:1 2023-10-04T00:33:02.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7 2 2023-10-04T00:33:02.686 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:02.682+0000 7ff1e591f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-04T00:33:02.818 INFO:teuthology.orchestra.run.smithi027.stdout:2 2023-10-04T00:33:02.836 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-04T00:33:02.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-04T00:33:02.838 INFO:tasks.ceph.osd.0:Started 2023-10-04T00:33:02.838 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-04T00:33:02.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-04T00:33:02.840 INFO:tasks.ceph.osd.1:Started 2023-10-04T00:33:02.841 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-04T00:33:02.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-04T00:33:02.843 INFO:tasks.ceph.osd.2:Started 2023-10-04T00:33:02.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:02.876 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:02.874+0000 7ff1e591f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-04T00:33:03.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:03.029 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:02.817302+0000","last_up_change":"0.000000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:03.044 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-04T00:33:03.044 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-04T00:33:03.345 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:03.474 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T00:33:03.474+0000 7fe2e92d4640 -1 Falling back to public interface 2023-10-04T00:33:03.487 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:33:03.486+0000 7fd4eb286640 -1 Falling back to public interface 2023-10-04T00:33:03.487 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:33:03.486+0000 7f29f2d2e640 -1 Falling back to public interface 2023-10-04T00:33:03.516 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.514+0000 7ff1e591f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-04T00:33:03.528 INFO:teuthology.misc.health.smithi027.stdout: 2023-10-04T00:33:03.528 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":5,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:02.817302+0000","last_up_change":"0.000000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:03.543 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-04T00:33:03.575 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.574+0000 7ff1e591f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-04T00:33:03.630 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.630+0000 7ff1e591f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-04T00:33:03.711 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.710+0000 7ff1e591f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-04T00:33:03.765 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.762+0000 7ff1e591f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-04T00:33:03.846 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.846+0000 7ff1e591f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-04T00:33:03.900 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:03.898+0000 7ff1e591f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-04T00:33:04.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:04.046+0000 7ff1e591f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-04T00:33:04.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:04.106+0000 7ff1e591f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-04T00:33:04.770 INFO:tasks.ceph.mgr.x.smithi027.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-04T00:33:04.771 INFO:tasks.ceph.mgr.x.smithi027.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-04T00:33:04.771 INFO:tasks.ceph.mgr.x.smithi027.stderr: from numpy import show_config as show_numpy_config 2023-10-04T00:33:04.837 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:04.834+0000 7ff1e591f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-04T00:33:04.969 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:04.966+0000 7ff1e591f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-04T00:33:05.127 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:05.126+0000 7ff1e591f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-04T00:33:05.195 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:05.194+0000 7ff1e591f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-04T00:33:05.325 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:05.322+0000 7ff1e591f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-04T00:33:05.949 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:05.946+0000 7ff1e591f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-04T00:33:06.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:06.030+0000 7ff1e591f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-04T00:33:06.221 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:06.218+0000 7ff1e591f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-04T00:33:06.299 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:06.298+0000 7ff1e591f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-04T00:33:06.630 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:06.630+0000 7ff1e591f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-04T00:33:06.694 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:06.694+0000 7ff1e591f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-04T00:33:06.865 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:33:06.862+0000 7f29f2d2e640 -1 osd.1 0 log_to_monitors true 2023-10-04T00:33:07.312 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T00:33:07.310+0000 7fe2e92d4640 -1 osd.2 0 log_to_monitors true 2023-10-04T00:33:07.371 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:33:07.370+0000 7fd4eb286640 -1 osd.0 0 log_to_monitors true 2023-10-04T00:33:07.933 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:33:07.930+0000 7ff18d192640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-04T00:33:09.845 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:10.345 INFO:teuthology.misc.health.smithi027.stdout: 2023-10-04T00:33:10.345 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":7,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:08.776886+0000","last_up_change":"0.000000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:10.370 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-04T00:33:10.587 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T00:33:10.586+0000 7fe2e4a59640 -1 osd.2 0 waiting for initial osdmap 2023-10-04T00:33:10.614 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T00:33:10.614+0000 7fe2dd0a0640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-04T00:33:10.627 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:33:10.626+0000 7fd4e8243640 -1 osd.0 0 waiting for initial osdmap 2023-10-04T00:33:10.637 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:33:10.638+0000 7f29ee4b3640 -1 osd.1 0 waiting for initial osdmap 2023-10-04T00:33:10.650 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:33:10.650+0000 7fd4df052640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-04T00:33:10.666 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:33:10.666+0000 7f29e6afa640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-04T00:33:16.672 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:16.968 INFO:teuthology.misc.health.smithi027.stdout: 2023-10-04T00:33:16.968 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":11,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:14.822695+0000","last_up_change":"2023-10-04T00:33:11.628861+0000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-04T00:33:12.795458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6811","nonce":2975557568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6815","nonce":2975557568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6821","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6823","nonce":2975557568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6817","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6819","nonce":2975557568}]},"public_addr":"172.21.15.27:6811/2975557568","cluster_addr":"172.21.15.27:6815/2975557568","heartbeat_back_addr":"172.21.15.27:6823/2975557568","heartbeat_front_addr":"172.21.15.27:6819/2975557568","state":["exists","up"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6810","nonce":3555328205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6814","nonce":3555328205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6822","nonce":3555328205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6818","nonce":3555328205}]},"public_addr":"172.21.15.27:6810/3555328205","cluster_addr":"172.21.15.27:6814/3555328205","heartbeat_back_addr":"172.21.15.27:6822/3555328205","heartbeat_front_addr":"172.21.15.27:6818/3555328205","state":["exists","up"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6801","nonce":3685667109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6803","nonce":3685667109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6807","nonce":3685667109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6805","nonce":3685667109}]},"public_addr":"172.21.15.27:6801/3685667109","cluster_addr":"172.21.15.27:6803/3685667109","heartbeat_back_addr":"172.21.15.27:6807/3685667109","heartbeat_front_addr":"172.21.15.27:6805/3685667109","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.399554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:07.881648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.271297+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:16.985 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-04T00:33:16.985 INFO:tasks.ceph:Creating RBD pool 2023-10-04T00:33:16.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-04T00:33:17.834 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2023-10-04T00:33:17.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-04T00:33:18.848 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-04T00:33:18.880 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-04T00:33:18.880 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-04T00:33:18.880 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-04T00:33:18.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-04T00:33:18.881 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-04T00:33:19.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:19.232 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1391102419},{"type":"v1","addr":"172.21.15.27:6825","nonce":1391102419}]},"active_addr":"172.21.15.27:6825/1391102419","active_change":"2023-10-04T00:33:06.715503+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2107599872}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1168916211}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1639844567}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1312794956}]}]} 2023-10-04T00:33:19.234 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-04T00:33:19.234 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-04T00:33:19.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:19.520 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:19.521 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:18.841297+0000","last_up_change":"2023-10-04T00:33:11.628861+0000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-04T00:33:12.795458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-04T00:33:17.315024+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6811","nonce":2975557568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6815","nonce":2975557568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6821","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6823","nonce":2975557568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6817","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6819","nonce":2975557568}]},"public_addr":"172.21.15.27:6811/2975557568","cluster_addr":"172.21.15.27:6815/2975557568","heartbeat_back_addr":"172.21.15.27:6823/2975557568","heartbeat_front_addr":"172.21.15.27:6819/2975557568","state":["exists","up"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6810","nonce":3555328205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6814","nonce":3555328205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6822","nonce":3555328205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6818","nonce":3555328205}]},"public_addr":"172.21.15.27:6810/3555328205","cluster_addr":"172.21.15.27:6814/3555328205","heartbeat_back_addr":"172.21.15.27:6822/3555328205","heartbeat_front_addr":"172.21.15.27:6818/3555328205","state":["exists","up"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6801","nonce":3685667109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6803","nonce":3685667109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6807","nonce":3685667109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6805","nonce":3685667109}]},"public_addr":"172.21.15.27:6801/3685667109","cluster_addr":"172.21.15.27:6803/3685667109","heartbeat_back_addr":"172.21.15.27:6807/3685667109","heartbeat_front_addr":"172.21.15.27:6805/3685667109","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.399554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:07.881648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.271297+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:19.538 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-04T00:33:19.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-04T00:33:19.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:19.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T00:33:18.841297+0000","last_up_change":"2023-10-04T00:33:11.628861+0000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-04T00:33:12.795458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-04T00:33:17.315024+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6811","nonce":2975557568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6815","nonce":2975557568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6821","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6823","nonce":2975557568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6817","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6819","nonce":2975557568}]},"public_addr":"172.21.15.27:6811/2975557568","cluster_addr":"172.21.15.27:6815/2975557568","heartbeat_back_addr":"172.21.15.27:6823/2975557568","heartbeat_front_addr":"172.21.15.27:6819/2975557568","state":["exists","up"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6810","nonce":3555328205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6814","nonce":3555328205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6822","nonce":3555328205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6818","nonce":3555328205}]},"public_addr":"172.21.15.27:6810/3555328205","cluster_addr":"172.21.15.27:6814/3555328205","heartbeat_back_addr":"172.21.15.27:6822/3555328205","heartbeat_front_addr":"172.21.15.27:6818/3555328205","state":["exists","up"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6801","nonce":3685667109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6803","nonce":3685667109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6807","nonce":3685667109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6805","nonce":3685667109}]},"public_addr":"172.21.15.27:6801/3685667109","cluster_addr":"172.21.15.27:6803/3685667109","heartbeat_back_addr":"172.21.15.27:6807/3685667109","heartbeat_front_addr":"172.21.15.27:6805/3685667109","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.399554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:07.881648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.271297+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T00:33:19.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-04T00:33:19.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-04T00:33:19.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-04T00:33:19.996 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2023-10-04T00:33:19.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-04T00:33:19.998 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2023-10-04T00:33:19.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-04T00:33:19.999 INFO:teuthology.orchestra.run.smithi027.stdout:34359738373 2023-10-04T00:33:20.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-04T00:33:20.308 INFO:teuthology.orchestra.run.smithi027.stdout:34359738371 2023-10-04T00:33:20.317 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2023-10-04T00:33:20.318 INFO:teuthology.orchestra.run.smithi027.stdout:34359738371 2023-10-04T00:33:20.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-10-04T00:33:20.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-04T00:33:20.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-10-04T00:33:21.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-04T00:33:21.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-04T00:33:21.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-04T00:33:21.646 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2023-10-04T00:33:21.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-10-04T00:33:21.669 DEBUG:teuthology.parallel:result is None 2023-10-04T00:33:21.670 INFO:teuthology.orchestra.run.smithi027.stdout:34359738373 2023-10-04T00:33:21.670 INFO:teuthology.orchestra.run.smithi027.stdout:34359738372 2023-10-04T00:33:21.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-04T00:33:21.690 DEBUG:teuthology.parallel:result is None 2023-10-04T00:33:21.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-10-04T00:33:21.691 DEBUG:teuthology.parallel:result is None 2023-10-04T00:33:21.691 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-04T00:33:21.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T00:33:21.999 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:22.000 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T00:33:22.016 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-04T00:33:20.724183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.888713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865890+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T00:33:18.865890+0000","last_peered":"2023-10-04T00:33:18.865890+0000","last_clean":"2023-10-04T00:33:18.865890+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T00:33:18.865890+0000","last_undegraded":"2023-10-04T00:33:18.865890+0000","last_fullsized":"2023-10-04T00:33:18.865890+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:56.745807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865570+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T00:33:18.865570+0000","last_peered":"2023-10-04T00:33:18.865570+0000","last_clean":"2023-10-04T00:33:18.865570+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T00:33:18.865570+0000","last_undegraded":"2023-10-04T00:33:18.865570+0000","last_fullsized":"2023-10-04T00:33:18.865570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:17:10.395923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.866816+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T00:33:18.866816+0000","last_peered":"2023-10-04T00:33:18.866816+0000","last_clean":"2023-10-04T00:33:18.866816+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T00:33:18.866816+0000","last_undegraded":"2023-10-04T00:33:18.866816+0000","last_fullsized":"2023-10-04T00:33:18.866816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:28:34.482874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865536+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T00:33:18.865536+0000","last_peered":"2023-10-04T00:33:18.865536+0000","last_clean":"2023-10-04T00:33:18.865536+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T00:33:18.865536+0000","last_undegraded":"2023-10-04T00:33:18.865536+0000","last_fullsized":"2023-10-04T00:33:18.865536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:00:33.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.868607+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T00:33:18.868607+0000","last_peered":"2023-10-04T00:33:18.868607+0000","last_clean":"2023-10-04T00:33:18.868607+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T00:33:18.868607+0000","last_undegraded":"2023-10-04T00:33:18.868607+0000","last_fullsized":"2023-10-04T00:33:18.868607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:51:26.823694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-04T00:33:19.857042+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T00:33:19.857042+0000","last_peered":"2023-10-04T00:33:19.857042+0000","last_clean":"2023-10-04T00:33:19.857042+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T00:33:19.857042+0000","last_undegraded":"2023-10-04T00:33:19.857042+0000","last_fullsized":"2023-10-04T00:33:19.857042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-04T00:33:19.857024+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T00:33:19.857024+0000","last_peered":"2023-10-04T00:33:19.857024+0000","last_clean":"2023-10-04T00:33:19.857024+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T00:33:19.857024+0000","last_undegraded":"2023-10-04T00:33:19.857024+0000","last_fullsized":"2023-10-04T00:33:19.857024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865486+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T00:33:18.865486+0000","last_peered":"2023-10-04T00:33:18.865486+0000","last_clean":"2023-10-04T00:33:18.865486+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T00:33:18.865486+0000","last_undegraded":"2023-10-04T00:33:18.865486+0000","last_fullsized":"2023-10-04T00:33:18.865486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:35:25.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-04T00:33:17.845310+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T00:33:17.845310+0000","last_peered":"2023-10-04T00:33:17.845310+0000","last_clean":"2023-10-04T00:33:17.845310+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T00:33:17.845310+0000","last_undegraded":"2023-10-04T00:33:17.845310+0000","last_fullsized":"2023-10-04T00:33:17.845310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T00:33:22.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T00:33:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:22.286 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T00:33:22.302 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-04T00:33:20.724183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.888713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865890+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T00:33:18.865890+0000","last_peered":"2023-10-04T00:33:18.865890+0000","last_clean":"2023-10-04T00:33:18.865890+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T00:33:18.865890+0000","last_undegraded":"2023-10-04T00:33:18.865890+0000","last_fullsized":"2023-10-04T00:33:18.865890+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:03:56.745807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865570+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T00:33:18.865570+0000","last_peered":"2023-10-04T00:33:18.865570+0000","last_clean":"2023-10-04T00:33:18.865570+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T00:33:18.865570+0000","last_undegraded":"2023-10-04T00:33:18.865570+0000","last_fullsized":"2023-10-04T00:33:18.865570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:17:10.395923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.866816+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T00:33:18.866816+0000","last_peered":"2023-10-04T00:33:18.866816+0000","last_clean":"2023-10-04T00:33:18.866816+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T00:33:18.866816+0000","last_undegraded":"2023-10-04T00:33:18.866816+0000","last_fullsized":"2023-10-04T00:33:18.866816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:28:34.482874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865536+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T00:33:18.865536+0000","last_peered":"2023-10-04T00:33:18.865536+0000","last_clean":"2023-10-04T00:33:18.865536+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T00:33:18.865536+0000","last_undegraded":"2023-10-04T00:33:18.865536+0000","last_fullsized":"2023-10-04T00:33:18.865536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:00:33.192227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.868607+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T00:33:18.868607+0000","last_peered":"2023-10-04T00:33:18.868607+0000","last_clean":"2023-10-04T00:33:18.868607+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T00:33:18.868607+0000","last_undegraded":"2023-10-04T00:33:18.868607+0000","last_fullsized":"2023-10-04T00:33:18.868607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:51:26.823694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-04T00:33:19.857042+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T00:33:19.857042+0000","last_peered":"2023-10-04T00:33:19.857042+0000","last_clean":"2023-10-04T00:33:19.857042+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T00:33:19.857042+0000","last_undegraded":"2023-10-04T00:33:19.857042+0000","last_fullsized":"2023-10-04T00:33:19.857042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-04T00:33:19.857024+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T00:33:19.857024+0000","last_peered":"2023-10-04T00:33:19.857024+0000","last_clean":"2023-10-04T00:33:19.857024+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T00:33:19.857024+0000","last_undegraded":"2023-10-04T00:33:19.857024+0000","last_fullsized":"2023-10-04T00:33:19.857024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-04T00:33:18.865486+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T00:33:18.865486+0000","last_peered":"2023-10-04T00:33:18.865486+0000","last_clean":"2023-10-04T00:33:18.865486+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T00:33:18.865486+0000","last_undegraded":"2023-10-04T00:33:18.865486+0000","last_fullsized":"2023-10-04T00:33:18.865486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:35:25.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-04T00:33:17.845310+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T00:33:17.845310+0000","last_peered":"2023-10-04T00:33:17.845310+0000","last_clean":"2023-10-04T00:33:17.845310+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T00:33:17.845310+0000","last_undegraded":"2023-10-04T00:33:17.845310+0000","last_fullsized":"2023-10-04T00:33:17.845310+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T00:33:22.304 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-04T00:33:22.304 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-04T00:33:22.304 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-04T00:33:22.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-04T00:33:22.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T00:33:22.627 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-04T00:33:22.646 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-04T00:33:22.647 INFO:teuthology.run_tasks:Running task exec... 2023-10-04T00:33:22.656 INFO:teuthology.task.exec:Executing custom commands... 2023-10-04T00:33:22.656 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T00:33:22.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-04T00:33:23.888 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'datapool' created 2023-10-04T00:33:23.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-04T00:33:26.942 INFO:teuthology.run_tasks:Running task install... 2023-10-04T00:33:26.950 DEBUG:teuthology.task.install:project ceph 2023-10-04T00:33:26.951 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-04T00:33:26.951 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-04T00:33:26.951 INFO:teuthology.task.install:Using flavor: default 2023-10-04T00:33:26.956 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-04T00:33:26.956 INFO:teuthology.task.install:extra packages: [] 2023-10-04T00:33:26.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-key list | grep Ceph 2023-10-04T00:33:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-04T00:33:27.049 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-04T00:33:27.050 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph.com (release key) 2023-10-04T00:33:27.050 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-04T00:33:27.050 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-04T00:33:27.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-04T00:33:27.258 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default/ 2023-10-04T00:33:27.259 INFO:teuthology.task.install.deb:Package version is 18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:33:27.370 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:33:27.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-04T00:33:27.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get update 2023-10-04T00:33:27.581 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-04T00:33:27.581 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-04T00:33:27.589 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-04T00:33:27.606 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-04T00:33:27.673 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy InRelease 2023-10-04T00:33:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:6 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release 2023-10-04T00:33:27.745 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-04T00:33:28.851 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:33:28.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6336-ga7de91bf-1jammy cephadm=18.0.0-6336-ga7de91bf-1jammy ceph-mds=18.0.0-6336-ga7de91bf-1jammy ceph-mgr=18.0.0-6336-ga7de91bf-1jammy ceph-common=18.0.0-6336-ga7de91bf-1jammy ceph-fuse=18.0.0-6336-ga7de91bf-1jammy ceph-test=18.0.0-6336-ga7de91bf-1jammy ceph-volume=18.0.0-6336-ga7de91bf-1jammy radosgw=18.0.0-6336-ga7de91bf-1jammy python3-rados=18.0.0-6336-ga7de91bf-1jammy python3-rgw=18.0.0-6336-ga7de91bf-1jammy python3-cephfs=18.0.0-6336-ga7de91bf-1jammy python3-rbd=18.0.0-6336-ga7de91bf-1jammy libcephfs2=18.0.0-6336-ga7de91bf-1jammy libcephfs-dev=18.0.0-6336-ga7de91bf-1jammy librados2=18.0.0-6336-ga7de91bf-1jammy librbd1=18.0.0-6336-ga7de91bf-1jammy rbd-fuse=18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:33:28.920 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:33:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2023-10-04T00:33:29.091 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-common is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-mds is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-mgr is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-test is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-volume is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-dev is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.243 INFO:teuthology.orchestra.run.smithi027.stdout:librados2 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:librbd1 is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:python3-cephfs is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:python3-rados is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:python3-rbd is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:python3-rgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:radosgw is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse is already the newest version (18.0.0-6336-ga7de91bf-1jammy). 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-04T00:33:29.244 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-04T00:33:29.245 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-04T00:33:29.245 INFO:teuthology.orchestra.run.smithi027.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-04T00:33:29.245 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-04T00:33:29.280 INFO:teuthology.orchestra.run.smithi027.stdout:0 upgraded, 0 newly installed, 0 to remove and 254 not upgraded. 2023-10-04T00:33:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-04T00:33:29.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-04T00:33:29.341 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2023-10-04T00:33:29.528 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2023-10-04T00:33:29.529 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2023-10-04T00:33:29.697 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2023-10-04T00:33:29.697 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-04T00:33:29.697 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-04T00:33:29.697 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-04T00:33:29.698 INFO:teuthology.orchestra.run.smithi027.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-04T00:33:29.698 INFO:teuthology.orchestra.run.smithi027.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-04T00:33:29.698 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-04T00:33:29.699 INFO:teuthology.orchestra.run.smithi027.stdout:The following additional packages will be installed: 2023-10-04T00:33:29.700 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemblk1 2023-10-04T00:33:29.701 INFO:teuthology.orchestra.run.smithi027.stdout:Suggested packages: 2023-10-04T00:33:29.701 INFO:teuthology.orchestra.run.smithi027.stdout: gnuplot gfio python-scipy 2023-10-04T00:33:29.728 INFO:teuthology.orchestra.run.smithi027.stdout:The following NEW packages will be installed: 2023-10-04T00:33:29.729 INFO:teuthology.orchestra.run.smithi027.stdout: fio libpmemblk1 2023-10-04T00:33:36.850 INFO:teuthology.orchestra.run.smithi027.stdout:0 upgraded, 2 newly installed, 0 to remove and 254 not upgraded. 2023-10-04T00:33:36.851 INFO:teuthology.orchestra.run.smithi027.stdout:Need to get 4,112 kB of archives. 2023-10-04T00:33:36.851 INFO:teuthology.orchestra.run.smithi027.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-04T00:33:36.851 INFO:teuthology.orchestra.run.smithi027.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-04T00:33:36.919 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-04T00:33:37.450 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 4,112 kB in 7s (561 kB/s) 2023-10-04T00:33:37.619 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-04T00:33:37.725 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172875 files and directories currently installed.) 2023-10-04T00:33:37.731 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-04T00:33:37.817 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-04T00:33:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package fio. 2023-10-04T00:33:38.190 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-04T00:33:38.228 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking fio (3.28-1) ... 2023-10-04T00:33:38.756 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-04T00:33:38.859 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up fio (3.28-1) ... 2023-10-04T00:33:39.387 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-04T00:33:40.479 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-04T00:33:40.980 INFO:teuthology.orchestra.run.smithi027.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-04T00:33:40.982 DEBUG:teuthology.parallel:result is None 2023-10-04T00:33:40.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-04T00:33:41.115 DEBUG:teuthology.orchestra.run.smithi027:> dpkg-query -W -f '${Version}' ceph 2023-10-04T00:33:41.135 INFO:teuthology.orchestra.run.smithi027.stdout:18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:33:41.135 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336-ga7de91bf-1jammy 2023-10-04T00:33:41.135 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336-ga7de91bf-1jammy is installed. 2023-10-04T00:33:41.135 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-04T00:33:41.135 INFO:teuthology.run_tasks:Running task workunit... 2023-10-04T00:33:41.147 INFO:tasks.workunit:Pulling workunits from ref 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 2023-10-04T00:33:41.147 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-04T00:33:41.147 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-04T00:33:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T00:33:41.182 INFO:teuthology.orchestra.run.smithi027.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-04T00:33:41.183 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-04T00:33:41.229 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-04T00:33:41.230 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-04T00:33:41.278 INFO:tasks.workunit:timeout=3h 2023-10-04T00:33:41.278 INFO:tasks.workunit:cleanup=True 2023-10-04T00:33:41.278 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ajarr/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 2023-10-04T00:33:41.328 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-04T00:34:18.923 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to '7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231'. 2023-10-04T00:34:18.923 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.923 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.924 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr: 2023-10-04T00:34:18.925 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at 7fa2fe7fd78 rbd_support_module_reacovery.sh: check CLIENT_ADDR non-zero 2023-10-04T00:34:18.929 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-04T00:34:18.978 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-04T00:34:18.981 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-04T00:34:18.981 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-04T00:34:19.716 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-04T00:34:19.800 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-04T00:34:19.860 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-04T00:34:19.862 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-04T00:34:19.862 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-04T00:34:19.916 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-04T00:34:19.919 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-04T00:34:19.920 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-04T00:34:19.969 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-04T00:34:19.970 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-04T00:34:19.970 DEBUG:teuthology.orchestra.run.smithi027:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-04T00:34:20.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ POOL=rbd 2023-10-04T00:34:20.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ IMAGE_PREFIX=image 2023-10-04T00:34:20.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ NUM_IMAGES=20 2023-10-04T00:34:20.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ RUN_TIME=3600 2023-10-04T00:34:20.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror pool enable rbd image 2023-10-04T00:34:20.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-04T00:34:20.111 INFO:tasks.workunit.client.0.smithi027.stdout:d15ecbdd-9949-45f0-9bc2-991d71fbbfb9 2023-10-04T00:34:20.114 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i = 1 )) 2023-10-04T00:34:20.114 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:20.114 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image1 2023-10-04T00:34:20.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-04T00:34:20.872 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:20.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-04T00:34:20.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:20.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:20.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image2 2023-10-04T00:34:21.005 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-04T00:34:21.884 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:21.900 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-04T00:34:21.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:21.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:21.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image3 2023-10-04T00:34:22.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-04T00:34:22.885 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:22.900 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-04T00:34:22.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:22.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:22.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image4 2023-10-04T00:34:23.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-04T00:34:23.897 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:23.914 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-04T00:34:23.978 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:23.978 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:23.978 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image5 2023-10-04T00:34:24.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-04T00:34:24.911 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:24.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-04T00:34:24.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:24.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:24.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image6 2023-10-04T00:34:25.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-04T00:34:25.918 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:25.934 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-04T00:34:25.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:25.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:25.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image7 2023-10-04T00:34:26.046 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-04T00:34:26.924 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:26.941 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-04T00:34:27.005 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:27.005 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:27.005 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image8 2023-10-04T00:34:27.054 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-04T00:34:27.928 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:27.944 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-04T00:34:28.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:28.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:28.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image9 2023-10-04T00:34:28.055 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-04T00:34:28.935 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:28.950 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-04T00:34:29.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:29.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:29.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image10 2023-10-04T00:34:29.064 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-04T00:34:29.951 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:29.969 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-04T00:34:30.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:30.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:30.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image11 2023-10-04T00:34:30.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-04T00:34:30.961 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:30.982 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-04T00:34:31.044 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:31.045 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:31.045 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image12 2023-10-04T00:34:31.092 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-04T00:34:31.953 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:31.968 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-04T00:34:32.030 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:32.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:32.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image13 2023-10-04T00:34:32.081 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-04T00:34:32.981 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:32.998 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-04T00:34:33.062 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:33.062 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:33.062 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image14 2023-10-04T00:34:33.109 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-04T00:34:33.979 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:33.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-04T00:34:34.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:34.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:34.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image15 2023-10-04T00:34:34.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-04T00:34:34.989 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:35.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-04T00:34:35.073 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:35.073 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:35.073 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image16 2023-10-04T00:34:35.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-04T00:34:36.006 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:36.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-04T00:34:36.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:36.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:36.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image17 2023-10-04T00:34:36.133 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-04T00:34:37.002 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:37.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-04T00:34:37.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:37.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:37.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image18 2023-10-04T00:34:37.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-04T00:34:38.009 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:38.026 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-04T00:34:38.089 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:38.089 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:38.089 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image19 2023-10-04T00:34:38.140 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-04T00:34:39.029 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:39.045 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-04T00:34:39.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:39.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:39.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd create --size 1G rbd/image20 2023-10-04T00:34:39.154 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-04T00:34:40.031 INFO:tasks.workunit.client.0.smithi027.stdout:Mirroring enabled 2023-10-04T00:34:40.046 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-04T00:34:40.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i = 1 )) 2023-10-04T00:34:40.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.107 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image1 2023-10-04T00:34:40.227 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-04T00:34:40.227 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.227 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.227 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.228 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image2 2023-10-04T00:34:40.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-04T00:34:40.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.300 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image3 2023-10-04T00:34:40.373 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-04T00:34:40.373 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.373 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.374 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.374 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image4 2023-10-04T00:34:40.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-04T00:34:40.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.450 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image5 2023-10-04T00:34:40.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-04T00:34:40.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.543 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image6 2023-10-04T00:34:40.614 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-04T00:34:40.614 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.614 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.615 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.615 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image7 2023-10-04T00:34:40.751 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-04T00:34:40.751 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.751 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.751 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.752 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image8 2023-10-04T00:34:40.885 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-04T00:34:40.885 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:40.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:40.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:40.886 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image9 2023-10-04T00:34:41.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-04T00:34:41.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.018 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image10 2023-10-04T00:34:41.184 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-04T00:34:41.185 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.185 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.185 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.185 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image11 2023-10-04T00:34:41.319 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-04T00:34:41.319 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.320 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image12 2023-10-04T00:34:41.440 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-04T00:34:41.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.441 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image13 2023-10-04T00:34:41.604 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-04T00:34:41.604 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.605 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.605 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.605 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image14 2023-10-04T00:34:41.712 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-04T00:34:41.712 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.712 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.715 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.726 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image15 2023-10-04T00:34:41.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-04T00:34:41.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:41.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:41.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:41.925 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image16 2023-10-04T00:34:42.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-04T00:34:42.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:42.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:42.079 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:42.079 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image17 2023-10-04T00:34:42.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-04T00:34:42.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:42.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:42.231 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image18 2023-10-04T00:34:42.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:42.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-04T00:34:42.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:42.441 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:42.441 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image19 2023-10-04T00:34:42.447 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:42.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-04T00:34:42.636 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:42.636 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:42.636 INFO:tasks.workunit.client.0.smithi027.stderr:++ sudo rbd device map rbd/image20 2023-10-04T00:34:42.636 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:42.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-04T00:34:42.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T00:34:42.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 20 )) 2023-10-04T00:34:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-04T00:34:42.840 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379682 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ END_TIME=1696383282 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR= 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR= 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379682 -le 1696383282 ]] 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z '' ]] 2023-10-04T00:34:42.842 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:34:43.706 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:34:43.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.699+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:34:43.994 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:34:43.994 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:34:43.994 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:34:43.995 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:34:43.995 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:34:43.995 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:34:43.995 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:34:43.995 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:34:43.997 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:43.987+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:34:48.037 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.031+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:34:48.037 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.031+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:34:48.037 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.031+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:34:48.330 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:34:48.330 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:34:48.330 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:34:48.330 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:34:48.330 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:34:48.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:34:48.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:34:48.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:34:48.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:34:48.323+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:34:52.847 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:34:52.847 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:34:52.850 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:34:52.854 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:34:53.833 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1639844567 2023-10-04T00:34:53.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:34:53.838 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379693 2023-10-04T00:34:53.838 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379693 -le 1696383282 ]] 2023-10-04T00:34:53.838 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1639844567 ]] 2023-10-04T00:34:53.838 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1639844567 != '' ]] 2023-10-04T00:34:53.838 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1639844567 2023-10-04T00:34:55.820 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1639844567 until 2023-10-04T01:34:55.061837+0000 (3600 sec) 2023-10-04T00:34:55.865 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:34:55.865 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1639844567 2023-10-04T00:34:56.642 INFO:tasks.workunit.client.0.smithi027.stderr:listed 1 entries 2023-10-04T00:34:56.674 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1639844567 2023-10-04T00:34:56.674 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:35:06.677 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:35:06.677 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:35:06.682 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:35:06.686 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:35:07.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1639844567 2023-10-04T00:35:07.586 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:35:07.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379707 2023-10-04T00:35:07.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379707 -le 1696383282 ]] 2023-10-04T00:35:07.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1639844567 ]] 2023-10-04T00:35:07.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1639844567 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\3\9\8\4\4\5\6\7 ]] 2023-10-04T00:35:07.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:35:17.599 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:35:17.601 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:35:17.606 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:35:17.624 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:35:18.500 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1639844567 2023-10-04T00:35:18.500 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:35:18.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379718 2023-10-04T00:35:18.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379718 -le 1696383282 ]] 2023-10-04T00:35:18.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1639844567 ]] 2023-10-04T00:35:18.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1639844567 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\3\9\8\4\4\5\6\7 ]] 2023-10-04T00:35:18.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:35:28.515 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:35:28.515 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:35:28.515 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:35:28.519 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:35:29.468 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:35:29.468 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:35:29.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379729 2023-10-04T00:35:29.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379729 -le 1696383282 ]] 2023-10-04T00:35:29.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:35:29.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\3\9\8\4\4\5\6\7 ]] 2023-10-04T00:35:29.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2713975836 2023-10-04T00:35:31.882 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2713975836 until 2023-10-04T01:35:31.095567+0000 (3600 sec) 2023-10-04T00:35:31.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2713975836 2023-10-04T00:35:31.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:35:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:listed 2 entries 2023-10-04T00:35:32.623 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:35:32.623 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:35:42.625 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:35:42.635 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:35:42.635 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:35:42.636 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:35:43.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:35:43.500 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:35:43.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379743 2023-10-04T00:35:43.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379743 -le 1696383282 ]] 2023-10-04T00:35:43.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:35:43.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:35:43.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:35:47.026 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:35:47.026 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:35:47.026 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:35:47.026 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:35:47.027 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:35:47.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:35:53.504 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:35:53.510 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:35:53.510 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:35:53.510 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:35:54.398 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:35:54.402 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:35:54.403 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379754 2023-10-04T00:35:54.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379754 -le 1696383282 ]] 2023-10-04T00:35:54.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:35:54.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:35:54.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.008+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.008+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c20480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.008+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.008+0000 7ff17e975640 -1 librbd::ImageState: 0x5644811e6980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.012+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.012+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.022 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.016+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.022 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.016+0000 7ff17e975640 -1 librbd::ImageState: 0x5644811e7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.020+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.020+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c20480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.024+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.024+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.028+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.028+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8ba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.036+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.036+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::ImageState: 0x564481451e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::ImageState: 0x564481450680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.044+0000 7ff17e975640 -1 librbd::ImageState: 0x5644811e7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.048+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.048+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c20000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.056+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.056+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8ba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.060+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.060+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.064+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.064+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.068+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.068+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.076+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.076+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c20c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.086 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.080+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.086 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.080+0000 7ff17e975640 -1 librbd::ImageState: 0x564481450500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.088+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.088+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.092+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:00.092+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8ba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:36:04.407 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:36:04.409 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:36:04.411 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:36:04.416 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:36:05.417 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:36:05.422 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:36:05.425 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379765 2023-10-04T00:36:05.425 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379765 -le 1696383282 ]] 2023-10-04T00:36:05.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:36:05.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:36:05.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:15.433 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:36:15.433 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:36:15.437 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:36:15.459 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:36:16.293 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:36:16.295 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:36:16.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379776 2023-10-04T00:36:16.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379776 -le 1696383282 ]] 2023-10-04T00:36:16.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:36:16.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:36:16.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:23.430 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:36:23.431 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:36:23.432 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:36:23.432 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:36:23.432 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:36:23.432 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:36:23.432 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:23.423+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:36:26.297 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:36:26.298 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:36:26.313 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:36:26.318 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:36:27.325 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2713975836 2023-10-04T00:36:27.325 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:36:27.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379787 2023-10-04T00:36:27.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379787 -le 1696383282 ]] 2023-10-04T00:36:27.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2713975836 ]] 2023-10-04T00:36:27.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2713975836 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:36:27.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:37.328 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:36:37.330 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:36:37.335 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:36:37.340 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:36:38.337 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:36:38.337 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:36:38.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379798 2023-10-04T00:36:38.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379798 -le 1696383282 ]] 2023-10-04T00:36:38.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:36:38.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\7\1\3\9\7\5\8\3\6 ]] 2023-10-04T00:36:38.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1592458368 2023-10-04T00:36:40.881 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1592458368 until 2023-10-04T01:36:40.122707+0000 (3600 sec) 2023-10-04T00:36:40.934 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:36:40.935 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1592458368 2023-10-04T00:36:41.779 INFO:tasks.workunit.client.0.smithi027.stderr:listed 3 entries 2023-10-04T00:36:41.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:36:41.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:51.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:36:51.844 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:36:51.853 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:36:51.854 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:36:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:36:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:36:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379812 2023-10-04T00:36:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379812 -le 1696383282 ]] 2023-10-04T00:36:52.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:36:52.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:36:52.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:36:58.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:36:58.633 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:36:58.633 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:36:58.633 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:36:58.633 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:36:58.626+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:59.998+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:36:59.998+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f86100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.002+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.002+0000 7ff17f977640 -1 librbd::ImageState: 0x564481450880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.002+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.006+0000 7ff17f977640 -1 librbd::ImageState: 0x5644811e7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.010+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.010+0000 7ff17f977640 -1 librbd::ImageState: 0x564481451580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.018+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.018+0000 7ff17f977640 -1 librbd::ImageState: 0x564481c20080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.022+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.022+0000 7ff17f977640 -1 librbd::ImageState: 0x56447adab000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.026+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.026+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f86480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.030+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.030+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f86300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.034+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.034+0000 7ff17f977640 -1 librbd::ImageState: 0x564481c21680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.046+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.046+0000 7ff17f977640 -1 librbd::ImageState: 0x564481d8bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.054+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.054+0000 7ff17f977640 -1 librbd::ImageState: 0x564481450500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.062+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.062+0000 7ff17f977640 -1 librbd::ImageState: 0x564481e1a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.066+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.066+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f86500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.070+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.070+0000 7ff17f977640 -1 librbd::ImageState: 0x5644813e7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.078+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.078+0000 7ff17f977640 -1 librbd::ImageState: 0x5644812d1d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.082+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.082+0000 7ff17f977640 -1 librbd::ImageState: 0x564481c21680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.086+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.094 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.086+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.090+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.090+0000 7ff17f977640 -1 librbd::ImageState: 0x564481450880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.094+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.094+0000 7ff17f977640 -1 librbd::ImageState: 0x564481d8bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.106 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.098+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:00.106 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:37:00.098+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f86000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:37:02.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:37:02.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:37:02.814 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:37:02.823 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:37:03.822 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:37:03.822 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:37:03.823 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379823 2023-10-04T00:37:03.824 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379823 -le 1696383282 ]] 2023-10-04T00:37:03.824 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:37:03.824 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:37:03.824 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:37:13.825 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:37:13.825 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:37:13.825 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:37:13.848 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:37:14.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:37:14.702 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:37:14.703 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379834 2023-10-04T00:37:14.703 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379834 -le 1696383282 ]] 2023-10-04T00:37:14.703 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:37:14.703 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:37:14.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:37:24.707 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:37:24.708 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:37:24.712 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:37:24.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:37:25.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:37:25.634 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:37:25.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379845 2023-10-04T00:37:25.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379845 -le 1696383282 ]] 2023-10-04T00:37:25.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:37:25.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:37:25.635 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:37:35.637 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:37:35.637 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:37:35.644 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:37:35.654 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:37:36.494 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1592458368 2023-10-04T00:37:36.494 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:37:36.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379856 2023-10-04T00:37:36.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379856 -le 1696383282 ]] 2023-10-04T00:37:36.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1592458368 ]] 2023-10-04T00:37:36.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1592458368 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:37:36.496 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:37:46.496 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:37:46.501 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:37:46.507 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:37:46.507 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:37:47.476 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:37:47.476 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:37:47.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379867 2023-10-04T00:37:47.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379867 -le 1696383282 ]] 2023-10-04T00:37:47.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:37:47.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\9\2\4\5\8\3\6\8 ]] 2023-10-04T00:37:47.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/706622636 2023-10-04T00:37:49.901 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/706622636 until 2023-10-04T01:37:49.176891+0000 (3600 sec) 2023-10-04T00:37:49.926 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/706622636 2023-10-04T00:37:49.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:37:50.669 INFO:tasks.workunit.client.0.smithi027.stderr:listed 4 entries 2023-10-04T00:37:50.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:37:50.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.001+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.001+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.005+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.005+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c20800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.009+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.009+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.013+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.013+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c21700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.021+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.021+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.029+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.029+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbf000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.033+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.033+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c21080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.037+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.037+0000 7ff17f176640 -1 librbd::ImageState: 0x5644813e7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.041+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.041+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.045+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.045+0000 7ff17e975640 -1 librbd::ImageState: 0x564481bbf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.053+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.053+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c20800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.057+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.057+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c20e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.061+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.061+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.065+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.065+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.069+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.073+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.077+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.077+0000 7ff17f176640 -1 librbd::ImageState: 0x564481d8a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.077+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.077+0000 7ff17f176640 -1 librbd::ImageState: 0x564481d8b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.081+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.081+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.085+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.085+0000 7ff17f176640 -1 librbd::ImageState: 0x564481450b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.085+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:38:00.085+0000 7ff17f176640 -1 librbd::ImageState: 0x564481451300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:38:00.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:00.714 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:00.717 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:00.718 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:01.623 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:38:01.624 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:01.624 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379881 2023-10-04T00:38:01.624 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379881 -le 1696383282 ]] 2023-10-04T00:38:01.625 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:38:01.625 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:01.625 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:38:03.779 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:03.773+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:38:11.626 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:11.627 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:11.627 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:11.635 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:12.554 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:38:12.556 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:12.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379892 2023-10-04T00:38:12.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379892 -le 1696383282 ]] 2023-10-04T00:38:12.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:38:12.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:12.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:22.559 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:22.559 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:22.561 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:22.573 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:23.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:38:23.509 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:23.510 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379903 2023-10-04T00:38:23.510 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379903 -le 1696383282 ]] 2023-10-04T00:38:23.510 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:38:23.510 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:23.510 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:33.511 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:33.516 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:33.517 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:33.531 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:34.031 INFO:tasks.workunit.client.0.smithi027.stderr:2023-10-04T00:38:34.029+0000 7f16c77fe640 0 --2- 172.21.15.27:0/14292611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c806f3a0 0x7f16c8120420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-04T00:38:34.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:38:34.734 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:34.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379914 2023-10-04T00:38:34.735 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379914 -le 1696383282 ]] 2023-10-04T00:38:34.735 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:38:34.735 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:34.735 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:36.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:38:36.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:38:36.676 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:38:36.676 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:38:36.677 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:38:36.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:38:44.740 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:44.740 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:44.750 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:44.754 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:45.649 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/706622636 2023-10-04T00:38:45.649 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:45.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379925 2023-10-04T00:38:45.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379925 -le 1696383282 ]] 2023-10-04T00:38:45.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/706622636 ]] 2023-10-04T00:38:45.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/706622636 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:45.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:38:55.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:38:55.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:38:55.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:38:55.657 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:38:56.590 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:38:56.590 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:38:56.591 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379936 2023-10-04T00:38:56.591 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379936 -le 1696383282 ]] 2023-10-04T00:38:56.591 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:38:56.591 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\7\0\6\6\2\2\6\3\6 ]] 2023-10-04T00:38:56.591 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2108081611 2023-10-04T00:38:58.916 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2108081611 until 2023-10-04T01:38:58.083656+0000 (3600 sec) 2023-10-04T00:38:58.956 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:38:58.956 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2108081611 2023-10-04T00:38:59.640 INFO:tasks.workunit.client.0.smithi027.stderr:listed 5 entries 2023-10-04T00:38:59.669 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:38:59.669 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:39:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.004+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564481d8be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.004+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.004+0000 7ff17f977640 -1 librbd::ImageState: 0x564481bbf700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.012+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.012+0000 7ff17f977640 -1 librbd::ImageState: 0x564481bbe380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.016+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.016+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbe600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.020+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.020+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.040+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.040+0000 7ff17f176640 -1 librbd::ImageState: 0x5644811e7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.044+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.044+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f86680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.048+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.048+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbf200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.056+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.056+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.060+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.060+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.072 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.068+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.072 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.068+0000 7ff17f176640 -1 librbd::ImageState: 0x5644811e7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.076+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.076+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f87a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.080+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.080+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f87c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.084+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.084+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f87e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.088+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.088+0000 7ff17f977640 -1 librbd::ImageState: 0x564482280080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.092+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.092+0000 7ff17f977640 -1 librbd::ImageState: 0x564482280280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.096+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.096+0000 7ff17f977640 -1 librbd::ImageState: 0x564482280480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.107 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.100+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.107 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.100+0000 7ff17f977640 -1 librbd::ImageState: 0x564482280e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.111 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.108+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:00.111 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:39:00.108+0000 7ff17f977640 -1 librbd::ImageState: 0x564482280d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:39:09.683 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:39:09.684 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:39:09.685 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:39:09.686 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:39:10.691 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:39:10.691 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:39:10.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379950 2023-10-04T00:39:10.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379950 -le 1696383282 ]] 2023-10-04T00:39:10.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:39:10.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:39:10.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:39:20.708 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:39:20.710 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:39:20.710 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:39:20.723 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:39:21.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:39:21.716 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:39:21.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379961 2023-10-04T00:39:21.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379961 -le 1696383282 ]] 2023-10-04T00:39:21.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:39:21.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:39:21.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:39:31.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:39:31.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:39:31.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:39:31.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:39:32.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:39:32.599 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:39:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379972 2023-10-04T00:39:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379972 -le 1696383282 ]] 2023-10-04T00:39:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:39:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:39:32.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:39:42.604 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:39:42.604 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:39:42.612 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:39:42.617 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:39:43.354 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:39:43.347+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:39:43.355 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:39:43.347+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:39:43.355 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:39:43.347+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:39:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:39:43.556 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:39:43.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379983 2023-10-04T00:39:43.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379983 -le 1696383282 ]] 2023-10-04T00:39:43.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:39:43.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:39:43.558 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:39:53.559 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:39:53.560 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:39:53.563 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:39:53.565 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:39:54.585 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2108081611 2023-10-04T00:39:54.585 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:39:54.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696379994 2023-10-04T00:39:54.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696379994 -le 1696383282 ]] 2023-10-04T00:39:54.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2108081611 ]] 2023-10-04T00:39:54.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2108081611 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:39:54.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:04.590 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:40:04.590 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:40:04.590 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:40:04.591 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:40:05.636 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:05.636 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:40:05.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380005 2023-10-04T00:40:05.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380005 -le 1696383282 ]] 2023-10-04T00:40:05.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:40:05.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\1\0\8\0\8\1\6\1\1 ]] 2023-10-04T00:40:05.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/4237805760 2023-10-04T00:40:07.948 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/4237805760 until 2023-10-04T01:40:07.178076+0000 (3600 sec) 2023-10-04T00:40:07.985 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/4237805760 2023-10-04T00:40:07.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:40:08.747 INFO:tasks.workunit.client.0.smithi027.stderr:listed 6 entries 2023-10-04T00:40:08.787 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:08.787 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:18.789 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:40:18.789 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:40:18.793 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:40:18.809 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:40:19.770 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:19.770 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:40:19.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380019 2023-10-04T00:40:19.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380019 -le 1696383282 ]] 2023-10-04T00:40:19.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:40:19.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:40:19.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:28.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:40:28.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:40:28.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:28.586+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:40:29.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:40:29.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:40:29.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:40:29.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:40:30.721 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:30.722 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:40:30.723 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380030 2023-10-04T00:40:30.723 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380030 -le 1696383282 ]] 2023-10-04T00:40:30.723 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:40:30.723 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:40:30.723 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:40.731 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:40:40.731 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:40:40.731 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:40:40.738 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:40:41.732 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:41.733 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:40:41.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380041 2023-10-04T00:40:41.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380041 -le 1696383282 ]] 2023-10-04T00:40:41.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:40:41.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:40:41.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:51.735 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:40:51.736 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:40:51.737 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:40:51.745 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:40:52.804 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:40:52.804 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:40:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380052 2023-10-04T00:40:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380052 -le 1696383282 ]] 2023-10-04T00:40:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:40:52.805 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:40:52.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:40:57.506 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:40:57.506 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:40:57.507 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:40:57.498+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:41:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:40:59.998+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:40:59.998+0000 7ff17f176640 -1 librbd::ImageState: 0x564482281480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.002+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.002+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.002+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.002+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.010+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.010+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c7c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.014+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.014+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7c900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.018+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.018+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.029 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.026+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.026+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c7d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.026+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.026+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c7cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.030+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.030+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.046+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.046+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.050+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.050+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.054+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.054+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.058+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.058+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.062+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.062+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.066+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.066+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.078+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.078+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.082+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.082+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.090+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.094+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.098+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.098+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.118 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.114+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:00.118 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:41:00.114+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:41:02.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:41:02.812 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:41:02.824 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:41:02.830 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:41:03.687 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4237805760 2023-10-04T00:41:03.687 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:41:03.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380063 2023-10-04T00:41:03.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380063 -le 1696383282 ]] 2023-10-04T00:41:03.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4237805760 ]] 2023-10-04T00:41:03.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4237805760 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:41:03.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:41:13.690 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:41:13.694 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:41:13.697 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:41:13.701 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:41:14.549 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380074 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380074 -le 1696383282 ]] 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\2\3\7\8\0\5\7\6\0 ]] 2023-10-04T00:41:14.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2013951676 2023-10-04T00:41:16.988 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2013951676 until 2023-10-04T01:41:16.144662+0000 (3600 sec) 2023-10-04T00:41:17.023 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:41:17.026 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2013951676 2023-10-04T00:41:17.774 INFO:tasks.workunit.client.0.smithi027.stderr:listed 7 entries 2023-10-04T00:41:17.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:41:17.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:41:27.818 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:41:27.821 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:41:27.823 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:41:27.833 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:41:28.759 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:41:28.759 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:41:28.760 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380088 2023-10-04T00:41:28.761 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380088 -le 1696383282 ]] 2023-10-04T00:41:28.761 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:41:28.761 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:41:28.761 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:41:38.767 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:41:38.768 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:41:38.768 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:41:38.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:41:39.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:41:39.733 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:41:39.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380099 2023-10-04T00:41:39.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380099 -le 1696383282 ]] 2023-10-04T00:41:39.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:41:39.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:41:39.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:41:49.752 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:41:49.757 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:41:49.757 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:41:49.773 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:41:50.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:41:50.677 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:41:50.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380110 2023-10-04T00:41:50.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380110 -le 1696383282 ]] 2023-10-04T00:41:50.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:41:50.679 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:41:50.679 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.001+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.001+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.005+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.005+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.009+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.009+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.013+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.013+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.021+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.021+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.029+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.029+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.041+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.041+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.061+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.061+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.061+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.061+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.073+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.073+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.081+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.081+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.089+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.089+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.093+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.093+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.093+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.093+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.099 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.097+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.099 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.097+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.107 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.105+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.105+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.116 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.113+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.116 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.113+0000 7ff17f977640 -1 librbd::ImageState: 0x564482444100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.120 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.117+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.120 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.117+0000 7ff17f977640 -1 librbd::ImageState: 0x564482444300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.126 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.125+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.126 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.125+0000 7ff17f977640 -1 librbd::ImageState: 0x564482444500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.130 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.129+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.130 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:42:00.129+0000 7ff17f977640 -1 librbd::ImageState: 0x564482444700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:42:00.680 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:00.680 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:00.692 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:00.693 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:01.593 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:42:01.594 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:01.606 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380121 2023-10-04T00:42:01.606 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380121 -le 1696383282 ]] 2023-10-04T00:42:01.606 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:42:01.606 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:42:01.606 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:11.611 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:11.611 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:11.613 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:11.614 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:12.481 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2013951676 2023-10-04T00:42:12.482 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:12.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380132 2023-10-04T00:42:12.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380132 -le 1696383282 ]] 2023-10-04T00:42:12.483 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2013951676 ]] 2023-10-04T00:42:12.483 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2013951676 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:42:12.483 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:22.487 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:22.488 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:22.489 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:22.498 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:23.444 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:42:23.444 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:23.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380143 2023-10-04T00:42:23.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380143 -le 1696383282 ]] 2023-10-04T00:42:23.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:42:23.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\1\3\9\5\1\6\7\6 ]] 2023-10-04T00:42:23.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1607046929 2023-10-04T00:42:25.981 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1607046929 until 2023-10-04T01:42:25.182808+0000 (3600 sec) 2023-10-04T00:42:26.026 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1607046929 2023-10-04T00:42:26.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:42:26.869 INFO:tasks.workunit.client.0.smithi027.stderr:listed 8 entries 2023-10-04T00:42:26.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:42:26.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:36.910 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:36.911 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:36.911 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:36.915 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:37.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:42:37.809 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:37.810 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380157 2023-10-04T00:42:37.810 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380157 -le 1696383282 ]] 2023-10-04T00:42:37.810 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:42:37.810 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:42:37.810 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:47.819 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:47.819 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:47.825 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:47.828 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:48.793 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:42:48.798 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:48.801 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380168 2023-10-04T00:42:48.801 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380168 -le 1696383282 ]] 2023-10-04T00:42:48.801 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:42:48.801 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:42:48.801 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:42:58.812 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:42:58.815 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:42:58.822 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:42:58.822 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:42:59.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:42:59.876 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:42:59.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380179 2023-10-04T00:42:59.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380179 -le 1696383282 ]] 2023-10-04T00:42:59.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:42:59.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:42:59.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:43:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.000+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.000+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239ef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.004+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.016+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.016+0000 7ff17e975640 -1 librbd::ImageState: 0x564482280a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.020+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.020+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7cd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.024+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.024+0000 7ff17e975640 -1 librbd::ImageState: 0x564482444d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.032+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.032+0000 7ff17e975640 -1 librbd::ImageState: 0x564482444c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.036+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.036+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.044+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.044+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.048+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.048+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.052+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.052+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.056+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.056+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.072+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.072+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.076+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.076+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.084+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.084+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.088+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.088+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.092+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.092+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.096+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.096+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.100+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.100+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.108+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.108+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.113 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.112+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:00.113 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:43:00.112+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:43:09.879 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:43:09.879 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:43:09.882 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:43:09.884 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:43:10.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:43:10.807 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:43:10.808 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380190 2023-10-04T00:43:10.808 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380190 -le 1696383282 ]] 2023-10-04T00:43:10.808 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:43:10.808 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:43:10.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:43:20.809 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:43:20.817 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:43:20.821 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:43:20.829 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:43:21.754 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:43:21.755 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:43:21.755 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380201 2023-10-04T00:43:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380201 -le 1696383282 ]] 2023-10-04T00:43:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:43:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:43:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:43:24.914 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:43:24.912+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:43:31.761 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:43:31.761 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:43:31.761 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:43:31.770 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:43:32.798 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1607046929 2023-10-04T00:43:32.799 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:43:32.799 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380212 2023-10-04T00:43:32.799 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380212 -le 1696383282 ]] 2023-10-04T00:43:32.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1607046929 ]] 2023-10-04T00:43:32.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1607046929 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:43:32.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:43:42.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:43:42.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:43:42.815 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:43:42.818 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:43:43.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:43:43.660 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:43:43.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380223 2023-10-04T00:43:43.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380223 -le 1696383282 ]] 2023-10-04T00:43:43.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:43:43.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\0\7\0\4\6\9\2\9 ]] 2023-10-04T00:43:43.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/928103956 2023-10-04T00:43:45.995 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/928103956 until 2023-10-04T01:43:45.239899+0000 (3600 sec) 2023-10-04T00:43:46.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:43:46.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/928103956 2023-10-04T00:43:46.815 INFO:tasks.workunit.client.0.smithi027.stderr:listed 9 entries 2023-10-04T00:43:46.840 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:43:46.840 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:43:56.846 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:43:56.846 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:43:56.859 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:43:56.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:43:57.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:43:57.769 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:43:57.770 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380237 2023-10-04T00:43:57.770 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380237 -le 1696383282 ]] 2023-10-04T00:43:57.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:43:57.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:43:57.771 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:44:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.004+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.004+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.008+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.008+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.016+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.016+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.020+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.020+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.024+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.024+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.028+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.032+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.036+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.036+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.041 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.040+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.041 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.040+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.044+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.044+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x564482546f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.064+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.064+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.076+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.076+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.084+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.084+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.088+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.088+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.096+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.097 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.096+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.100+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.100+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.106 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.104+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.106 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.104+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.108+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:00.110 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:00.108+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258ed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:44:07.775 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:44:07.778 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:44:07.779 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:44:07.792 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:44:08.854 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:44:08.855 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:44:08.855 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380248 2023-10-04T00:44:08.855 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380248 -le 1696383282 ]] 2023-10-04T00:44:08.855 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:44:08.855 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:44:08.856 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:44:18.857 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:44:18.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:44:18.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:44:18.871 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:44:19.818 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:44:19.818 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:44:19.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380259 2023-10-04T00:44:19.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380259 -le 1696383282 ]] 2023-10-04T00:44:19.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:44:19.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:44:19.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:44:29.820 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:44:29.821 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:44:29.822 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:44:29.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:44:30.745 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:44:30.746 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:44:30.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380270 2023-10-04T00:44:30.747 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380270 -le 1696383282 ]] 2023-10-04T00:44:30.747 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:44:30.747 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:44:30.747 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:44:40.753 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:44:40.759 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:44:40.762 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:44:40.771 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:44:41.740 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/928103956 2023-10-04T00:44:41.751 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:44:41.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380281 2023-10-04T00:44:41.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380281 -le 1696383282 ]] 2023-10-04T00:44:41.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/928103956 ]] 2023-10-04T00:44:41.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/928103956 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:44:41.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:44:51.759 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:44:51.759 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:44:51.768 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:44:51.768 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:44:52.745 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:44:52.745 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:44:52.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380292 2023-10-04T00:44:52.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380292 -le 1696383282 ]] 2023-10-04T00:44:52.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:44:52.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\2\8\1\0\3\9\5\6 ]] 2023-10-04T00:44:52.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2079221933 2023-10-04T00:44:54.979 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:44:54.979 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:44:54.979 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:44:54.979 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:44:54.980 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:44:54.981 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:44:54.971+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:44:54.991 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2079221933 until 2023-10-04T01:44:54.284144+0000 (3600 sec) 2023-10-04T00:44:55.049 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:44:55.049 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2079221933 2023-10-04T00:44:55.824 INFO:tasks.workunit.client.0.smithi027.stderr:listed 10 entries 2023-10-04T00:44:55.883 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:44:55.883 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:45:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:59.999+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:44:59.999+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.003+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.003+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.007+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.007+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.011+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.011+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.015+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.015+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.019+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.019+0000 7ff17e975640 -1 librbd::ImageState: 0x564482281680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.023+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.023+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.027+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.027+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.031+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.031 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.031+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.035+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.035+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.039+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.039+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.043+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.043+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.047+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.047+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.051+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.051+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.051+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.051+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.063+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.063+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.075+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.075+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.075+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.075+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.083+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.083+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.087+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:45:00.087+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825c7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:45:05.019 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:45:05.019+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:45:05.019 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:45:05.019+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:45:05.019 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:45:05.019+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:45:05.884 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:45:05.887 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:45:05.889 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:45:05.899 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:45:06.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:45:06.782 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:45:06.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380306 2023-10-04T00:45:06.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380306 -le 1696383282 ]] 2023-10-04T00:45:06.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:45:06.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:45:06.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:45:16.789 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:45:16.790 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:45:16.796 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:45:16.799 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:45:17.636 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:45:17.636 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:45:17.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380317 2023-10-04T00:45:17.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380317 -le 1696383282 ]] 2023-10-04T00:45:17.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:45:17.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:45:17.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:45:27.647 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:45:27.647 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:45:27.648 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:45:27.658 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:45:28.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:45:28.540 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:45:28.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380328 2023-10-04T00:45:28.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380328 -le 1696383282 ]] 2023-10-04T00:45:28.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:45:28.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:45:28.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:45:38.542 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:45:38.545 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:45:38.550 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:45:38.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:45:39.575 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:45:39.582 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:45:39.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380339 2023-10-04T00:45:39.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380339 -le 1696383282 ]] 2023-10-04T00:45:39.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:45:39.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:45:39.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:45:49.584 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:45:49.584 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:45:49.595 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:45:49.602 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:45:50.449 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2079221933 2023-10-04T00:45:50.449 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:45:50.449 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380350 2023-10-04T00:45:50.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380350 -le 1696383282 ]] 2023-10-04T00:45:50.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2079221933 ]] 2023-10-04T00:45:50.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2079221933 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:45:50.450 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:00.451 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:00.451 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:00.453 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:00.454 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:01.596 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:01.596 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:01.597 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380361 2023-10-04T00:46:01.597 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380361 -le 1696383282 ]] 2023-10-04T00:46:01.597 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:01.597 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\7\9\2\2\1\9\3\3 ]] 2023-10-04T00:46:01.598 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/978409279 2023-10-04T00:46:03.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.078+0000 7ff17f977640 -1 librbd::managed_lock::GetLockerRequest: 0x564481e11730 handle_get_lockers: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.078+0000 7ff17f977640 -1 librbd::managed_lock::AcquireRequest: 0x5644828530e0 handle_get_locker: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.078+0000 7ff17f977640 -1 librbd::ManagedLock: 0x564481984698 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.078+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1a00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.078+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x56448285f340 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.218 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.218+0000 7ff17f176640 -1 librbd::ImageWatcher: 0x564481b84900 image watch failed: 94852247379968, (107) Transport endpoint is not connected 2023-10-04T00:46:03.219 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.218+0000 7ff17f176640 -1 librbd::Watcher: 0x564481b84900 handle_error: handle=94852247379968: (107) Transport endpoint is not connected 2023-10-04T00:46:03.230 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.230+0000 7ff17f176640 -1 librbd::managed_lock::BreakRequest: 0x5644825e4c80 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.230 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.230+0000 7ff17f176640 -1 librbd::managed_lock::AcquireRequest: 0x5644828523c0 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.230 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.230+0000 7ff17f176640 -1 librbd::ManagedLock: 0x564481392018 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.230 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.230+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x5644828205b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.230 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.230+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x56448282b760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:46:03.531 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:46:03.530+0000 7ff17f176640 -1 librbd::watcher::RewatchRequest: 0x5644814cc730 handle_unwatch client blocklisted 2023-10-04T00:46:04.060 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/978409279 until 2023-10-04T01:46:03.342576+0000 (3600 sec) 2023-10-04T00:46:04.097 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:46:04.097 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/978409279 2023-10-04T00:46:04.899 INFO:tasks.workunit.client.0.smithi027.stderr:listed 11 entries 2023-10-04T00:46:04.941 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:04.941 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:07.117 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:07.114+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:46:07.118 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:07.114+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:46:14.943 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:14.948 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:14.951 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:14.963 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:15.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:15.996 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:15.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380376 2023-10-04T00:46:15.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380376 -le 1696383282 ]] 2023-10-04T00:46:15.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:15.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:46:15.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:19.499 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:46:19.490+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:46:23.351 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:23.346+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:46:23.351 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:23.346+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:46:23.351 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:23.346+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:46:23.351 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:23.346+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:46:25.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:26.000 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:26.004 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:26.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:26.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:26.996 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:26.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380387 2023-10-04T00:46:26.996 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380387 -le 1696383282 ]] 2023-10-04T00:46:26.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:26.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:46:26.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:30.443 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:46:30.438+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:46:30.443 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:46:30.438+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:46:30.443 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:46:30.438+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:46:37.007 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:37.009 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:37.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:37.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:37.905 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:37.905 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:37.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380397 2023-10-04T00:46:37.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380397 -le 1696383282 ]] 2023-10-04T00:46:37.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:37.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:46:37.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:39.869 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:39.862+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:46:39.869 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:46:39.862+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:46:47.907 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:47.917 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:47.920 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:47.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:48.988 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:48.988 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:48.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380408 2023-10-04T00:46:48.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380408 -le 1696383282 ]] 2023-10-04T00:46:48.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:48.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:46:48.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:46:58.994 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:46:58.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:46:58.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:46:58.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:46:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/978409279 2023-10-04T00:46:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:46:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380419 2023-10-04T00:46:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380419 -le 1696383282 ]] 2023-10-04T00:46:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/978409279 ]] 2023-10-04T00:46:59.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/978409279 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:46:59.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:47:05.601 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.598+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:47:05.606 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:47:05.607 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:47:05.607 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:47:05.607 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:05.602+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:47:09.963 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:47:09.976 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:47:09.977 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:47:09.981 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:47:10.972 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:10.973 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:47:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380430 2023-10-04T00:47:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380430 -le 1696383282 ]] 2023-10-04T00:47:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:47:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\9\7\8\4\0\9\2\7\9 ]] 2023-10-04T00:47:10.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2080516480 2023-10-04T00:47:11.437 INFO:tasks.workunit.client.0.smithi027.stderr:2023-10-04T00:47:11.442+0000 7fa65effd640 0 --2- 172.21.15.27:0/1275798759 >> v2:172.21.15.27:3300/0 conn(0x7fa6580b2130 0x7fa6580b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-04T00:47:13.025 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2080516480 until 2023-10-04T01:47:12.214035+0000 (3600 sec) 2023-10-04T00:47:13.077 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:47:13.077 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2080516480 2023-10-04T00:47:13.924 INFO:tasks.workunit.client.0.smithi027.stderr:listed 12 entries 2023-10-04T00:47:13.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:13.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:47:20.680 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:47:20.682+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:47:20.680 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:47:20.682+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:47:20.680 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:47:20.682+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:47:23.955 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:47:23.955 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:47:23.955 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:47:23.957 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:47:25.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:25.027 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:47:25.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380445 2023-10-04T00:47:25.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380445 -le 1696383282 ]] 2023-10-04T00:47:25.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:47:25.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:47:25.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:47:35.035 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:47:35.036 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:47:35.064 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:47:35.064 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:47:35.985 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:35.985 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:47:35.994 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380456 2023-10-04T00:47:35.994 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380456 -le 1696383282 ]] 2023-10-04T00:47:35.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:47:35.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:47:35.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:47:46.004 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:47:46.007 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:47:46.011 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:47:46.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:47:46.920 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:46.920 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:47:46.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380466 2023-10-04T00:47:46.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380466 -le 1696383282 ]] 2023-10-04T00:47:46.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:47:46.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:47:46.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:47:47.374 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:47.377+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:47:47.374 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:47.377+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:47:47.375 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:47.377+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:47:47.375 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:47.377+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:47:48.449 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.449+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:47:48.450 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.453+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:47:48.450 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.453+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:47:48.450 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.453+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:47:48.450 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.453+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:47:48.450 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:47:48.453+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:47:56.933 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:47:56.933 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:47:56.933 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:47:56.951 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:47:57.860 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:47:57.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:47:57.860 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380477 2023-10-04T00:47:57.861 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380477 -le 1696383282 ]] 2023-10-04T00:47:57.861 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:47:57.861 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:47:57.861 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:48:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.001+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.001+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.009+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.009+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.013+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.013+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.017+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.017+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.041+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.041+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.045+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.045+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.049+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.049+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.053+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.053+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.065+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.065+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.065+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.065+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.077+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.077+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.076 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.081+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.081+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.085+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.085+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.089+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.089+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.093+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.093+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.101+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.101+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.109+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.109+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.112 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.113+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.112 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.113+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.117 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.121+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.117 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.121+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.122 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.125+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.122 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:48:00.125+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:48:00.851 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:48:00.853+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:48:00.851 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:48:00.853+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:48:00.851 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:48:00.853+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:48:07.863 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:48:07.864 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:48:07.866 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:48:07.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:48:08.768 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2080516480 2023-10-04T00:48:08.768 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:48:08.768 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380488 2023-10-04T00:48:08.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380488 -le 1696383282 ]] 2023-10-04T00:48:08.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2080516480 ]] 2023-10-04T00:48:08.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2080516480 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:48:08.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:48:18.774 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:48:18.774 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:48:18.776 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:48:18.780 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:48:19.791 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:48:19.791 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:48:19.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380499 2023-10-04T00:48:19.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380499 -le 1696383282 ]] 2023-10-04T00:48:19.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:48:19.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\0\8\0\5\1\6\4\8\0 ]] 2023-10-04T00:48:19.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3301872187 2023-10-04T00:48:22.061 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3301872187 until 2023-10-04T01:48:21.336122+0000 (3600 sec) 2023-10-04T00:48:22.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:48:22.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3301872187 2023-10-04T00:48:22.818 INFO:tasks.workunit.client.0.smithi027.stderr:listed 13 entries 2023-10-04T00:48:22.856 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:48:22.856 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:48:27.178 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:48:27.177+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:48:32.858 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:48:32.858 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:48:32.859 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:48:32.861 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:48:33.725 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:48:33.725 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:48:33.727 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380513 2023-10-04T00:48:33.728 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380513 -le 1696383282 ]] 2023-10-04T00:48:33.728 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:48:33.728 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:48:33.728 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:48:36.544 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:48:36.545+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:48:43.729 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:48:43.730 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:48:43.730 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:48:43.734 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:48:44.740 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:48:44.740 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:48:44.741 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380524 2023-10-04T00:48:44.741 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380524 -le 1696383282 ]] 2023-10-04T00:48:44.741 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:48:44.741 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:48:44.742 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:48:54.750 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:48:54.750 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:48:54.751 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:48:54.775 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:48:55.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:48:55.717 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:48:55.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380535 2023-10-04T00:48:55.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380535 -le 1696383282 ]] 2023-10-04T00:48:55.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:48:55.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:48:55.719 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:49:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::ImageState: 0x5644824e4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::ImageState: 0x56448239eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.004+0000 7ff17f977640 -1 librbd::ImageState: 0x56448239f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::ImageState: 0x56448239f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::ImageState: 0x56448239f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.008+0000 7ff17f977640 -1 librbd::ImageState: 0x56448258f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.012+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.012+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235bc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.012+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.012+0000 7ff17f977640 -1 librbd::ImageState: 0x56448258f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.016+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.016+0000 7ff17f977640 -1 librbd::ImageState: 0x5644825e1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.020+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.020+0000 7ff17f977640 -1 librbd::ImageState: 0x5644811e6c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.022 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.024+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.024+0000 7ff17f977640 -1 librbd::ImageState: 0x564481f87280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.028+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.028+0000 7ff17f176640 -1 librbd::ImageState: 0x56448235bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.040+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.040+0000 7ff17f977640 -1 librbd::ImageState: 0x5644825e1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.044+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.044+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.048+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.048+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d4800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.056+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.056+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.060+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.060+0000 7ff17f176640 -1 librbd::ImageState: 0x564482444600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.064+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.064+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d4780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.064+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:49:00.064+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d4380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:49:05.723 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:49:05.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:49:05.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:49:05.728 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:49:06.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:49:06.676 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:49:06.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380546 2023-10-04T00:49:06.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380546 -le 1696383282 ]] 2023-10-04T00:49:06.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:49:06.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:49:06.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:49:16.682 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:49:16.683 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:49:16.684 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:49:16.692 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:49:17.652 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3301872187 2023-10-04T00:49:17.653 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:49:17.653 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380557 2023-10-04T00:49:17.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380557 -le 1696383282 ]] 2023-10-04T00:49:17.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3301872187 ]] 2023-10-04T00:49:17.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3301872187 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:49:17.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:49:27.661 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:49:27.661 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:49:27.662 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:49:27.664 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:49:28.663 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:49:28.663 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:49:28.664 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380568 2023-10-04T00:49:28.664 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380568 -le 1696383282 ]] 2023-10-04T00:49:28.664 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:49:28.664 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\3\0\1\8\7\2\1\8\7 ]] 2023-10-04T00:49:28.664 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3859805389 2023-10-04T00:49:31.068 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3859805389 until 2023-10-04T01:49:30.299296+0000 (3600 sec) 2023-10-04T00:49:31.121 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:49:31.121 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3859805389 2023-10-04T00:49:31.887 INFO:tasks.workunit.client.0.smithi027.stderr:listed 14 entries 2023-10-04T00:49:31.943 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:49:31.944 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:49:41.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:49:41.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:49:41.946 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:49:41.947 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:49:42.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:49:42.835 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:49:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380582 2023-10-04T00:49:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380582 -le 1696383282 ]] 2023-10-04T00:49:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:49:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:49:42.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:49:52.838 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:49:52.839 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:49:52.839 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:49:52.846 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:49:53.730 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:49:53.730+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:49:53.730 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:49:53.730+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:49:53.761 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:49:53.766 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:49:53.776 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380593 2023-10-04T00:49:53.776 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380593 -le 1696383282 ]] 2023-10-04T00:49:53.777 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:49:53.777 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:49:53.777 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:50:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.006+0000 7ff17e975640 -1 librbd::ImageState: 0x564482444980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.010+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.010+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.018+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.018+0000 7ff17e975640 -1 librbd::ImageState: 0x564482281900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.022 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.022+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.022 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.022+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.026+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.026+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.034+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.034+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d5680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.038+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.038+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.038+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.038+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.042+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.042+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.046+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.046+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d4a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.046+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.050+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.050+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.050+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.054+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.054+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.056 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.054+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.056 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.054+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.058+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.058+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.066+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.066+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.070+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.070+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.074+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:50:00.074+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:50:03.788 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:50:03.789 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:50:03.789 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:50:03.800 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:50:04.799 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:50:04.801 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:50:04.821 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380604 2023-10-04T00:50:04.821 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380604 -le 1696383282 ]] 2023-10-04T00:50:04.821 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:50:04.821 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:50:04.821 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:50:12.752 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:50:12.750+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:50:14.858 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:50:14.859 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:50:14.859 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:50:14.879 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:50:15.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:50:15.794 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:50:15.795 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380615 2023-10-04T00:50:15.795 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380615 -le 1696383282 ]] 2023-10-04T00:50:15.795 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:50:15.796 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:50:15.796 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:50:25.356 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:50:25.357 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:25.358+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:50:25.883 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:50:25.883 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:50:25.890 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:50:25.890 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:50:26.857 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:50:26.857 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:50:26.858 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380626 2023-10-04T00:50:26.858 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380626 -le 1696383282 ]] 2023-10-04T00:50:26.858 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:50:26.858 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:50:26.858 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:50:35.229 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:50:35.233+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:50:35.229 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:50:35.233+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:50:35.229 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:50:35.233+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:50:36.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:50:36.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:50:36.872 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:50:36.883 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:50:37.512 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:50:37.512 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:50:37.512 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:50:37.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:50:37.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3859805389 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380637 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380637 -le 1696383282 ]] 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3859805389 ]] 2023-10-04T00:50:37.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3859805389 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:50:37.876 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:50:47.877 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:50:47.878 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:50:47.885 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:50:47.885 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:50:48.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:50:48.809 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:50:48.815 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380648 2023-10-04T00:50:48.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380648 -le 1696383282 ]] 2023-10-04T00:50:48.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:50:48.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\5\9\8\0\5\3\8\9 ]] 2023-10-04T00:50:48.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1678579581 2023-10-04T00:50:51.105 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1678579581 until 2023-10-04T01:50:50.316122+0000 (3600 sec) 2023-10-04T00:50:51.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:50:51.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1678579581 2023-10-04T00:50:51.884 INFO:tasks.workunit.client.0.smithi027.stderr:listed 15 entries 2023-10-04T00:50:51.930 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:50:51.930 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.005+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.005+0000 7ff17f176640 -1 librbd::ImageState: 0x564482280880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.005+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.005+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.009+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.009+0000 7ff17f176640 -1 librbd::ImageState: 0x56448235bd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.013+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.013+0000 7ff17f176640 -1 librbd::ImageState: 0x564482547e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.017+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.021+0000 7ff17f176640 -1 librbd::ImageState: 0x56448235bc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.033+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.033+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.041+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.041+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.045+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.045+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825c7d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.053+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.053+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.057+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.057+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.061+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.061+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.073+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.073+0000 7ff17f176640 -1 librbd::ImageState: 0x564481d8b680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.077+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.077+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.076 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.081+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.076 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.081+0000 7ff17f176640 -1 librbd::ImageState: 0x564481bbfa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.081+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.081+0000 7ff17f176640 -1 librbd::ImageState: 0x564481e1a180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.085+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.085+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.089+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.089+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.093+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.093+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e4280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.093+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.093+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.097+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:00.097+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e4580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:01.944 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:01.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:01.950 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:01.952 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:02.895 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:51:02.896 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:02.896 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380662 2023-10-04T00:51:02.896 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380662 -le 1696383282 ]] 2023-10-04T00:51:02.897 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:51:02.897 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:02.897 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:12.906 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:12.908 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:12.908 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:12.913 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:13.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:51:13.835 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:13.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380673 2023-10-04T00:51:13.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380673 -le 1696383282 ]] 2023-10-04T00:51:13.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:51:13.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:13.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:23.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:23.841 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:23.843 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:23.846 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:25.159 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:51:25.160 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:25.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380685 2023-10-04T00:51:25.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380685 -le 1696383282 ]] 2023-10-04T00:51:25.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:51:25.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:25.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:33.115 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:51:33.116+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:51:33.115 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:51:33.116+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:51:33.115 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:51:33.116+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:51:35.168 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:35.172 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:35.172 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:35.189 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:36.117 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:51:36.117 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:36.117 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380696 2023-10-04T00:51:36.118 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380696 -le 1696383282 ]] 2023-10-04T00:51:36.118 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:51:36.118 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:36.118 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:46.118 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:46.124 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:46.128 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:46.131 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:47.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1678579581 2023-10-04T00:51:47.007 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:47.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380707 2023-10-04T00:51:47.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380707 -le 1696383282 ]] 2023-10-04T00:51:47.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1678579581 ]] 2023-10-04T00:51:47.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1678579581 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:47.009 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:51:57.012 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:51:57.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:51:57.024 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:51:57.032 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:51:57.976 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:51:57.976 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:51:57.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380717 2023-10-04T00:51:57.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380717 -le 1696383282 ]] 2023-10-04T00:51:57.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:51:57.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\6\7\8\5\7\9\5\8\1 ]] 2023-10-04T00:51:57.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1057421862 2023-10-04T00:51:59.998 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:59.999+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:51:59.999 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:51:59.999+0000 7ff17e975640 -1 librbd::ImageState: 0x5644811e6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.003+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.003+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c20b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.011+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.011+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.015+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.015+0000 7ff17e975640 -1 librbd::ImageState: 0x564481bbf600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.023+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.023+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.027+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.027+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.035+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.035+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.039+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.039+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f86180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.051+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.051+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.059+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.059+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.067+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.067+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.071+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.071+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.075+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.075+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e0a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.091+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.091 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.091+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.095+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.095+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.103+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.101 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.103+0000 7ff17f176640 -1 librbd::ImageState: 0x5644813e7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.103 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1057421862 until 2023-10-04T01:51:59.366405+0000 (3600 sec) 2023-10-04T00:52:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.111+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.111+0000 7ff17f176640 -1 librbd::ImageState: 0x564482444880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.115 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.115+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.115 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.115+0000 7ff17f176640 -1 librbd::ImageState: 0x564482445c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.120 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.123+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.121 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.123+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.129 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.131+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.130 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:52:00.131+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:52:00.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:52:00.159 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1057421862 2023-10-04T00:52:00.817 INFO:tasks.workunit.client.0.smithi027.stderr:listed 16 entries 2023-10-04T00:52:00.850 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:00.850 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:08.899 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.899+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:52:08.906 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:52:08.906 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:52:08.906 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:52:08.906 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:52:08.906 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:52:08.907 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:52:08.907 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:52:08.907 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:52:08.907 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:08.903+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:52:10.856 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:52:10.856 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:52:10.858 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:52:10.866 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:52:11.901 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:11.901 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:52:11.902 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380731 2023-10-04T00:52:11.902 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380731 -le 1696383282 ]] 2023-10-04T00:52:11.902 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:52:11.903 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:52:11.903 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:21.904 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:52:21.904 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:52:21.906 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:52:21.909 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:52:22.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:22.954 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:52:22.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380742 2023-10-04T00:52:22.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380742 -le 1696383282 ]] 2023-10-04T00:52:22.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:52:22.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:52:22.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:28.446 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:28.447+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:52:28.447 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:28.447+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:52:31.083 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.083+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:52:31.087 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:52:31.088 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:52:31.089 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:52:31.087+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:52:32.956 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:52:32.957 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:52:32.957 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:52:32.960 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:52:33.213 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:33.215+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:52:33.214 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:33.215+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:52:33.214 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:33.215+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:52:34.021 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:34.021 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:52:34.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380754 2023-10-04T00:52:34.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380754 -le 1696383282 ]] 2023-10-04T00:52:34.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:52:34.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:52:34.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:44.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:52:44.035 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:52:44.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:52:44.047 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:52:45.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:45.099 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:52:45.100 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380765 2023-10-04T00:52:45.100 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380765 -le 1696383282 ]] 2023-10-04T00:52:45.100 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:52:45.101 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:52:45.101 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:47.187 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:47.186+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:52:47.188 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:47.186+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:52:47.188 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:47.186+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:52:55.102 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:52:55.103 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:52:55.104 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:52:55.127 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:52:56.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1057421862 2023-10-04T00:52:56.002 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:52:56.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380776 2023-10-04T00:52:56.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380776 -le 1696383282 ]] 2023-10-04T00:52:56.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1057421862 ]] 2023-10-04T00:52:56.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1057421862 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:52:56.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:52:57.905 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:52:57.906+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:53:04.797 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.790+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:53:04.797 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.794+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:53:04.800 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:53:04.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:53:04.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:53:04.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:53:04.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:04.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:53:06.004 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:53:06.005 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:53:06.006 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:53:06.006 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:53:06.978 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:06.978 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:53:06.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380786 2023-10-04T00:53:06.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380786 -le 1696383282 ]] 2023-10-04T00:53:06.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:53:06.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\0\5\7\4\2\1\8\6\2 ]] 2023-10-04T00:53:06.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/361536328 2023-10-04T00:53:09.115 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/361536328 until 2023-10-04T01:53:08.352318+0000 (3600 sec) 2023-10-04T00:53:09.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:53:09.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/361536328 2023-10-04T00:53:10.063 INFO:tasks.workunit.client.0.smithi027.stderr:listed 17 entries 2023-10-04T00:53:10.083 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:10.083 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:53:11.350 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:11.342+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:53:13.472 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:13.474+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:53:13.472 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:13.474+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:53:13.472 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:13.474+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:53:13.473 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:13.474+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:53:20.085 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:53:20.085 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:53:20.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:53:20.088 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:53:21.054 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:21.058 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:53:21.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380801 2023-10-04T00:53:21.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380801 -le 1696383282 ]] 2023-10-04T00:53:21.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:53:21.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:53:21.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:53:23.971 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:53:23.971 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:53:23.971 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:53:23.972 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:53:23.972 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:53:23.972 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:53:23.972 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:53:23.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:53:30.187 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:30.194+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:53:30.189 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:53:30.194+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:53:31.061 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:53:31.061 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:53:31.061 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:53:31.063 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:53:32.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:32.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:53:32.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380812 2023-10-04T00:53:32.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380812 -le 1696383282 ]] 2023-10-04T00:53:32.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:53:32.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:53:32.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:53:42.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:53:42.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:53:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:53:42.042 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:53:43.001 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:43.001 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:53:43.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380823 2023-10-04T00:53:43.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380823 -le 1696383282 ]] 2023-10-04T00:53:43.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:53:43.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:53:43.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:53:53.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:53:53.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:53:53.023 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:53:53.026 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:53:53.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:53:53.927 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:53:53.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380833 2023-10-04T00:53:53.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380833 -le 1696383282 ]] 2023-10-04T00:53:53.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:53:53.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:53:53.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:03.932 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:54:03.939 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:54:03.947 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:54:03.962 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:54:04.865 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/361536328 2023-10-04T00:54:04.865 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:54:04.866 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380844 2023-10-04T00:54:04.866 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380844 -le 1696383282 ]] 2023-10-04T00:54:04.867 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/361536328 ]] 2023-10-04T00:54:04.867 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/361536328 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:54:04.867 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:14.872 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:54:14.872 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:54:14.876 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:54:14.887 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:54:15.891 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:54:15.891 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:54:15.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380855 2023-10-04T00:54:15.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380855 -le 1696383282 ]] 2023-10-04T00:54:15.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:54:15.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\1\5\3\6\3\2\8 ]] 2023-10-04T00:54:15.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/303862463 2023-10-04T00:54:16.156 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:54:16.157+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:54:18.154 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/303862463 until 2023-10-04T01:54:17.377016+0000 (3600 sec) 2023-10-04T00:54:18.203 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/303862463 2023-10-04T00:54:18.223 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:54:19.029 INFO:tasks.workunit.client.0.smithi027.stderr:listed 18 entries 2023-10-04T00:54:19.061 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:54:19.061 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:25.746 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.749+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:54:25.755 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:54:25.756 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:25.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:54:27.574 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:27.573+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:54:27.575 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:27.573+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:54:29.063 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:54:29.063 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:54:29.066 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:54:29.071 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:54:30.033 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:54:30.033 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:54:30.033 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:54:30.033 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:54:30.033 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:54:30.034 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:30.037+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:54:30.039 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:54:30.044 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:54:30.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380870 2023-10-04T00:54:30.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380870 -le 1696383282 ]] 2023-10-04T00:54:30.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:54:30.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:54:30.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:40.049 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:54:40.050 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:54:40.050 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:54:40.055 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:54:41.063 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:54:41.064 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:54:41.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380881 2023-10-04T00:54:41.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380881 -le 1696383282 ]] 2023-10-04T00:54:41.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:54:41.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:54:41.066 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:43.085 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:43.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:54:43.086 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:43.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:54:43.086 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:54:43.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:54:51.068 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:54:51.071 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:54:51.080 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:54:51.086 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:54:51.979 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:54:51.979 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:54:51.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380891 2023-10-04T00:54:51.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380891 -le 1696383282 ]] 2023-10-04T00:54:51.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:54:51.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:54:51.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:54:59.999 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.004+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:54:59.999 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.004+0000 7ff17e975640 -1 librbd::ImageState: 0x56448239ea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.004+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.004+0000 7ff17e975640 -1 librbd::ImageState: 0x564482444300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.008+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.008+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825e1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.016+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.016+0000 7ff17e975640 -1 librbd::ImageState: 0x564481c7d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.020+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.020+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825e0000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.028+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.028+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.032+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.032+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d4780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.036+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.036+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.044+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.044+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.048+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.048+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.052+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.052+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258fa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.056+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.056+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d5680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.060+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.060+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.064+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.064+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e4700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.072+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.072+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e5f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.076+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.072 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.076+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.076+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.076+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.080+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.080+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.088+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.088+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.092+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:55:00.092+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258e180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:01.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:01.983 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:01.986 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:01.989 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:02.867 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:55:02.867 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:55:02.868 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380902 2023-10-04T00:55:02.868 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380902 -le 1696383282 ]] 2023-10-04T00:55:02.868 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:55:02.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:55:02.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:55:06.332 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:06.328+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:55:06.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:06.348+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:55:06.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:06.348+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:55:06.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:06.348+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:55:12.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:12.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:12.875 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:12.877 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:13.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/303862463 2023-10-04T00:55:13.871 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:55:13.871 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380913 2023-10-04T00:55:13.871 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380913 -le 1696383282 ]] 2023-10-04T00:55:13.872 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/303862463 ]] 2023-10-04T00:55:13.872 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/303862463 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:55:13.872 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:55:23.876 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:23.877 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:23.878 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:23.896 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:24.910 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:55:24.910 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:55:24.911 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380924 2023-10-04T00:55:24.911 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380924 -le 1696383282 ]] 2023-10-04T00:55:24.911 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:55:24.911 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\0\3\8\6\2\4\6\3 ]] 2023-10-04T00:55:24.911 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1264225453 2023-10-04T00:55:26.731 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1264225453 until 2023-10-04T01:55:26.006070+0000 (3600 sec) 2023-10-04T00:55:26.766 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1264225453 2023-10-04T00:55:26.766 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:55:27.489 INFO:tasks.workunit.client.0.smithi027.stderr:listed 19 entries 2023-10-04T00:55:27.523 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:55:27.524 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:55:27.577 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.580+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:55:27.577 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.580+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:55:27.577 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.580+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:55:27.582 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.584+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:55:27.582 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.584+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:55:27.582 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:27.584+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:55:36.809 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:36.812+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:55:36.810 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:36.812+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:55:37.525 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:37.530 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:37.538 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:37.545 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:38.498 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:55:38.498 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:55:38.498 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380938 2023-10-04T00:55:38.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380938 -le 1696383282 ]] 2023-10-04T00:55:38.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:55:38.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:55:38.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:55:48.504 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:48.504 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:48.505 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:48.515 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:49.629 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:55:49.629 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:55:49.629 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380949 2023-10-04T00:55:49.630 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380949 -le 1696383282 ]] 2023-10-04T00:55:49.630 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:55:49.630 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:55:49.630 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:55:56.377 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:56.379+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:55:56.388 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:56.391+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:55:56.389 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:55:56.391+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:55:59.632 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:55:59.632 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:55:59.632 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:55:59.635 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:55:59.998 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.003+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:55:59.999 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.003+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825c7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.011+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.011+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.019+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.019+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e0800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.031+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.031+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c7d000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.029 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.031+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.029 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.031+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.035+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.035+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.035+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.035+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.039+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.039+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239fa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.043+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.043+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.043+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.043+0000 7ff17f176640 -1 librbd::ImageState: 0x564481f87a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.047+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.047+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.051+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.051+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.055+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.055+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.063+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.063+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.067+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.067+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258fa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.079+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.079+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.083+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.083+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.087+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.087+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.103+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.103+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.103 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.107+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.103 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:56:00.107+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:00.561 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:56:00.562 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:00.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380960 2023-10-04T00:56:00.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380960 -le 1696383282 ]] 2023-10-04T00:56:00.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:56:00.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:56:00.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:10.573 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:56:10.574 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:56:10.575 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:56:10.582 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:56:11.496 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:56:11.496 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:11.496 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380971 2023-10-04T00:56:11.496 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380971 -le 1696383282 ]] 2023-10-04T00:56:11.496 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:56:11.497 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:56:11.497 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:11.783 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:56:11.783+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:56:21.498 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:56:21.502 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:56:21.510 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:56:21.514 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:56:22.432 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1264225453 2023-10-04T00:56:22.433 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:22.434 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380982 2023-10-04T00:56:22.434 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380982 -le 1696383282 ]] 2023-10-04T00:56:22.434 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1264225453 ]] 2023-10-04T00:56:22.434 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1264225453 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:56:22.434 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:25.412 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:56:25.414+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:56:25.413 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:56:25.414+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:56:27.019 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:56:27.018+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:56:32.435 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:56:32.436 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:56:32.446 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:56:32.458 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:56:33.151 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:56:33.154+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:56:33.151 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:56:33.154+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:56:33.574 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:56:33.578 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:33.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696380993 2023-10-04T00:56:33.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696380993 -le 1696383282 ]] 2023-10-04T00:56:33.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:56:33.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\2\6\4\2\2\5\4\5\3 ]] 2023-10-04T00:56:33.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/4009918629 2023-10-04T00:56:35.210 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/4009918629 until 2023-10-04T01:56:34.396497+0000 (3600 sec) 2023-10-04T00:56:35.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:56:35.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/4009918629 2023-10-04T00:56:35.842 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:56:35.842+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:56:35.975 INFO:tasks.workunit.client.0.smithi027.stderr:listed 20 entries 2023-10-04T00:56:36.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:56:36.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:37.257 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:56:37.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:56:46.014 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:56:46.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:56:46.016 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:56:46.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:56:46.994 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:56:46.994 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:46.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381007 2023-10-04T00:56:46.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381007 -le 1696383282 ]] 2023-10-04T00:56:46.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:56:46.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:56:46.995 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:56.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:56:56.999 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:56:57.001 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:56:57.005 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:56:57.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:56:57.980 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:56:57.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381017 2023-10-04T00:56:57.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381017 -le 1696383282 ]] 2023-10-04T00:56:57.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:56:57.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:56:57.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:56:59.997 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.001+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:56:59.997 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.001+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.005+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.000 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.005+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825e0900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.013+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.013+0000 7ff17e975640 -1 librbd::ImageState: 0x564481e1bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.017+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.017+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825e0400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.021+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.021+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d4800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.029+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.029+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.037+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.037+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.044 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.049+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.044 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.049+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.053+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.049 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.053+0000 7ff17e975640 -1 librbd::ImageState: 0x564481451680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.057+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.057+0000 7ff17e975640 -1 librbd::ImageState: 0x56448235b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.069+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.069+0000 7ff17e975640 -1 librbd::ImageState: 0x56448235bf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.069+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.069+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.076 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.081+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.076 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.081+0000 7ff17e975640 -1 librbd::ImageState: 0x564482280200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.081 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.085+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.085+0000 7ff17e975640 -1 librbd::ImageState: 0x564482280c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.089+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.089+0000 7ff17e975640 -1 librbd::ImageState: 0x564482281c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.093+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.093+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829ae080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.109+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.109+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829ae280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.118 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.121+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.118 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.121+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829ae480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.123 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.129+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.123 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.129+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.127 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.133+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:00.127 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:57:00.133+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:57:07.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:57:07.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:57:07.984 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:57:07.984 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:57:09.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:57:09.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:57:09.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381029 2023-10-04T00:57:09.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381029 -le 1696383282 ]] 2023-10-04T00:57:09.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:57:09.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:57:09.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:57:19.031 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:57:19.031 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:57:19.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:57:19.055 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:57:20.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:57:20.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:57:20.087 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381040 2023-10-04T00:57:20.087 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381040 -le 1696383282 ]] 2023-10-04T00:57:20.088 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:57:20.088 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:57:20.088 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:57:30.098 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:57:30.102 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:57:30.106 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:57:30.117 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:57:31.032 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:57:31.032 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:57:31.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381051 2023-10-04T00:57:31.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381051 -le 1696383282 ]] 2023-10-04T00:57:31.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:57:31.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:57:31.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:57:41.042 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:57:41.042 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:57:41.042 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:57:41.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:57:42.097 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/4009918629 2023-10-04T00:57:42.097 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:57:42.098 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381062 2023-10-04T00:57:42.098 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381062 -le 1696383282 ]] 2023-10-04T00:57:42.098 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/4009918629 ]] 2023-10-04T00:57:42.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/4009918629 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:57:42.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:57:49.922 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:57:49.923+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:57:49.922 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:57:49.923+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:57:49.922 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:57:49.923+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:57:52.100 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:57:52.100 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:57:52.101 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:57:52.101 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:57:53.119 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:57:53.119 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:57:53.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381073 2023-10-04T00:57:53.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381073 -le 1696383282 ]] 2023-10-04T00:57:53.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:57:53.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\4\0\0\9\9\1\8\6\2\9 ]] 2023-10-04T00:57:53.120 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1535402043 2023-10-04T00:57:55.202 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1535402043 until 2023-10-04T01:57:54.419965+0000 (3600 sec) 2023-10-04T00:57:55.244 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:57:55.244 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1535402043 2023-10-04T00:57:56.014 INFO:tasks.workunit.client.0.smithi027.stderr:listed 21 entries 2023-10-04T00:57:56.039 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:57:56.039 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:58:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.002+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.002+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.001 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.002+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.002+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262e800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.006+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.006+0000 7ff17f176640 -1 librbd::ImageState: 0x564482444500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.010+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.010+0000 7ff17f176640 -1 librbd::ImageState: 0x564481c21f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.014+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.014+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825e0b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.022+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.022+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.022+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.022+0000 7ff17f176640 -1 librbd::ImageState: 0x5644824e5380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.026+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.026+0000 7ff17f176640 -1 librbd::ImageState: 0x56448239f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.042+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.042+0000 7ff17f977640 -1 librbd::ImageState: 0x56448239ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.041 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.042+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.041 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.042+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.046+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.047 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.046+0000 7ff17f176640 -1 librbd::ImageState: 0x5644825d4200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.054+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.054+0000 7ff17f176640 -1 librbd::ImageState: 0x5644811e7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.058+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.058+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829ae380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.068 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.066+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.068 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.070+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829ae600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.070+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.070+0000 7ff17f176640 -1 librbd::ImageState: 0x56448258ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.082+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.082+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829ae800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.082+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.082+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829aea00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.094+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.094+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829aec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.098+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.096 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.098+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829af900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.110+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:00.109 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:58:00.110+0000 7ff17f176640 -1 librbd::ImageState: 0x5644829af780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:58:05.814 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:05.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:58:06.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:58:06.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:58:06.050 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:58:06.052 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:58:07.102 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:58:07.103 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:58:07.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381087 2023-10-04T00:58:07.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381087 -le 1696383282 ]] 2023-10-04T00:58:07.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:58:07.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:58:07.104 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:58:17.106 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:58:17.106 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:58:17.110 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:58:17.112 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:58:18.070 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:58:18.070 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:58:18.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381098 2023-10-04T00:58:18.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381098 -le 1696383282 ]] 2023-10-04T00:58:18.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:58:18.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:58:18.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:58:22.982 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:22.981+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:58:22.983 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:22.981+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:58:28.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:58:28.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:58:28.076 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:58:28.095 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:58:29.125 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:58:29.125 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:58:29.126 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381109 2023-10-04T00:58:29.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381109 -le 1696383282 ]] 2023-10-04T00:58:29.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:58:29.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:58:29.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:58:31.402 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:58:31.405+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:58:39.128 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:58:39.131 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:58:39.132 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:58:39.151 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:58:39.997 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:58:39.998 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:58:40.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381120 2023-10-04T00:58:40.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381120 -le 1696383282 ]] 2023-10-04T00:58:40.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:58:40.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:58:40.003 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:58:48.075 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.076+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:58:48.085 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:58:48.086 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:58:48.086 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:58:48.087 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:58:48.087 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:58:48.088+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:58:50.025 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:58:50.025 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:58:50.025 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:58:50.040 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:58:50.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1535402043 2023-10-04T00:58:50.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:58:50.989 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381130 2023-10-04T00:58:50.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381130 -le 1696383282 ]] 2023-10-04T00:58:50.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1535402043 ]] 2023-10-04T00:58:50.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1535402043 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:58:50.990 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:01.004 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:59:01.005 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:01.005 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:01.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:02.169 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:02.169 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:59:02.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381142 2023-10-04T00:59:02.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381142 -le 1696383282 ]] 2023-10-04T00:59:02.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T00:59:02.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\3\5\4\0\2\0\4\3 ]] 2023-10-04T00:59:02.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1990656485 2023-10-04T00:59:03.285 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.283+0000 7ff17f176640 -1 librbd::ImageWatcher: 0x564482695500 image watch failed: 94852248115200, (107) Transport endpoint is not connected 2023-10-04T00:59:03.285 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.283+0000 7ff17f176640 -1 librbd::Watcher: 0x564482695500 handle_error: handle=94852248115200: (107) Transport endpoint is not connected 2023-10-04T00:59:03.291 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.291+0000 7ff17f176640 -1 librbd::watcher::RewatchRequest: 0x564481b18a50 handle_unwatch client blocklisted 2023-10-04T00:59:03.291 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.291+0000 7ff17f176640 -1 librbd::Watcher: 0x564482695500 handle_rewatch: client blocklisted 2023-10-04T00:59:03.318 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.319+0000 7ff17e975640 -1 librbd::managed_lock::GetLockerRequest: 0x564481e101c0 handle_get_lockers: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.319 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.319+0000 7ff17e975640 -1 librbd::managed_lock::AcquireRequest: 0x564482c13590 handle_get_locker: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.319 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.319+0000 7ff17e975640 -1 librbd::ManagedLock: 0x56448139da18 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.319 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.319+0000 7ff17e975640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564482820dd0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.319 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.319+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x56448294cb00 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.325 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.323+0000 7ff17e975640 -1 librbd::ImageWatcher: 0x564482b9c900 image watch failed: 94852251308032, (107) Transport endpoint is not connected 2023-10-04T00:59:03.325 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.323+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9c900 handle_error: handle=94852251308032: (107) Transport endpoint is not connected 2023-10-04T00:59:03.338 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::ImageWatcher: 0x564482b9cf00 image watch failed: 94852248113152, (107) Transport endpoint is not connected 2023-10-04T00:59:03.338 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9cf00 handle_error: handle=94852248113152: (107) Transport endpoint is not connected 2023-10-04T00:59:03.338 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::watcher::RewatchRequest: 0x564481e68960 handle_unwatch client blocklisted 2023-10-04T00:59:03.339 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9c900 handle_rewatch: client blocklisted 2023-10-04T00:59:03.339 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::ManagedLock: 0x564481ae4018 send_reacquire_lock: aborting reacquire due to invalid watch handle 2023-10-04T00:59:03.339 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::watcher::RewatchRequest: 0x564481e689b0 handle_unwatch client blocklisted 2023-10-04T00:59:03.339 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9cf00 handle_rewatch: client blocklisted 2023-10-04T00:59:03.340 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::ManagedLock: 0x564481364358 send_acquire_lock: blocklisted waiting for exclusive lock 2023-10-04T00:59:03.340 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::ManagedLock: 0x564481364358 send_acquire_lock: watcher not registered - client blocklisted 2023-10-04T00:59:03.340 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564482820ea0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.340 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.339+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x5644826978c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:03.352 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::ImageWatcher: 0x564482694c00 image watch failed: 94852251311104, (107) Transport endpoint is not connected 2023-10-04T00:59:03.352 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::Watcher: 0x564482694c00 handle_error: handle=94852251311104: (107) Transport endpoint is not connected 2023-10-04T00:59:03.352 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::ImageWatcher: 0x564482b9d200 image watch failed: 94852248110080, (107) Transport endpoint is not connected 2023-10-04T00:59:03.352 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9d200 handle_error: handle=94852248110080: (107) Transport endpoint is not connected 2023-10-04T00:59:03.353 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::watcher::RewatchRequest: 0x564481e79770 handle_unwatch client blocklisted 2023-10-04T00:59:03.353 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::Watcher: 0x564482694c00 handle_rewatch: client blocklisted 2023-10-04T00:59:03.353 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::watcher::RewatchRequest: 0x56448205d6d0 handle_unwatch client blocklisted 2023-10-04T00:59:03.353 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:03.351+0000 7ff17e975640 -1 librbd::Watcher: 0x564482b9d200 handle_rewatch: client blocklisted 2023-10-04T00:59:04.277 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1990656485 until 2023-10-04T01:59:03.577087+0000 (3600 sec) 2023-10-04T00:59:04.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T00:59:04.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1990656485 2023-10-04T00:59:04.332 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.331+0000 7ff17e975640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.332 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.331+0000 7ff17e975640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564482821040 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.332 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.331+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x564482bfa840 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.445 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:04.447+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:04.445 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:04.447+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:04.572 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.571+0000 7ff17e975640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.572 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.571+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x5644826978c0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.572 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.571+0000 7ff17e975640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482a95f00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.572 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.571+0000 7ff17e975640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.592 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.591+0000 7ff17e975640 -1 librbd::managed_lock::ReleaseRequest: 0x5644821182a0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.789 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.787+0000 7ff17f176640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.789 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.791+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.790 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.791+0000 7ff17f176640 -1 librbd::managed_lock::ReleaseRequest: 0x5644821215c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.790 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.791+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc0b60 handle_refresh_image: failed to refresh image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.790 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.791+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482697760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.888 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.887+0000 7ff17e975640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.889 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.887+0000 7ff17e975640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.889 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.887+0000 7ff17e975640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564482820d00 handle_refresh_image: failed to refresh image: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.889 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.887+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x5644828e8dc0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:04.890 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:04.891+0000 7ff17e975640 -1 librbd::managed_lock::ReleaseRequest: 0x564482002120 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:05.308 INFO:tasks.workunit.client.0.smithi027.stderr:listed 22 entries 2023-10-04T00:59:05.340 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:05.341 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:06.297 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.295+0000 7ff17f176640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.297 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.295+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564482820f70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.298 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.295+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482bfbe40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.298 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.299+0000 7ff17e975640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.298 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.299+0000 7ff17e975640 -1 librbd::io::AioCompletion: 0x564482697760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.299 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.299+0000 7ff17e975640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482bdd340 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.310 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.311+0000 7ff17e975640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.310 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.311+0000 7ff17e975640 -1 librbd::managed_lock::ReleaseRequest: 0x56448161a6c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T00:59:06.311 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T00:59:06.311+0000 7ff17e975640 -1 librbd::ManagedLock: 0x564481ae4018 send_acquire_lock: watcher not registered - client blocklisted 2023-10-04T00:59:07.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.331+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:07.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.331+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:07.332 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.331+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:07.332 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.331+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:07.333 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.335+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:07.334 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:07.335+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:14.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:59:14.714 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:14.714 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:14.714 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:14.714 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:59:14.715 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:14.715 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:14.715 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:14.711+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:15.342 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:59:15.343 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:15.349 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:15.356 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:16.358 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:16.359 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:59:16.359 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381156 2023-10-04T00:59:16.359 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381156 -le 1696383282 ]] 2023-10-04T00:59:16.359 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T00:59:16.360 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T00:59:16.360 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:18.503 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:18.503 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:18.504 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:18.504 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:18.504 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:18.504 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:18.499+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:20.537 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:59:20.539+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T00:59:20.538 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:59:20.539+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T00:59:20.538 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T00:59:20.539+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T00:59:26.361 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:26.362 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:26.366 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:26.369 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:59:27.430 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:27.431 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:59:27.431 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381167 2023-10-04T00:59:27.431 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381167 -le 1696383282 ]] 2023-10-04T00:59:27.432 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T00:59:27.432 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T00:59:27.432 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:32.159 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.158+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:59:32.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:32.166 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:32.166 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:32.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:37.447 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:37.449 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:59:37.449 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:37.456 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:38.477 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:38.479 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:59:38.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381178 2023-10-04T00:59:38.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381178 -le 1696383282 ]] 2023-10-04T00:59:38.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T00:59:38.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T00:59:38.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:47.081 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:47.082 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:47.082 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:47.082 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:47.082 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:47.082 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:47.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:48.487 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:48.487 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T00:59:48.494 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:48.496 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:49.194 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.189+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T00:59:49.194 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.189+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T00:59:49.194 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.193+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T00:59:49.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.209+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T00:59:49.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.209+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T00:59:49.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.209+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T00:59:49.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.209+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T00:59:49.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T00:59:49.209+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T00:59:49.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T00:59:49.507 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T00:59:49.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381189 2023-10-04T00:59:49.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381189 -le 1696383282 ]] 2023-10-04T00:59:49.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T00:59:49.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T00:59:49.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T00:59:59.511 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T00:59:59.513 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T00:59:59.514 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T00:59:59.542 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:00.488 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1990656485 2023-10-04T01:00:00.488 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:00.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381200 2023-10-04T01:00:00.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381200 -le 1696383282 ]] 2023-10-04T01:00:00.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1990656485 ]] 2023-10-04T01:00:00.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1990656485 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T01:00:00.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:00:10.491 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:10.491 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:00:10.495 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:00:10.510 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:00:11.468 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:11.469 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:11.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381211 2023-10-04T01:00:11.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381211 -le 1696383282 ]] 2023-10-04T01:00:11.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:00:11.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\9\9\0\6\5\6\4\8\5 ]] 2023-10-04T01:00:11.470 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2670833392 2023-10-04T01:00:13.266 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2670833392 until 2023-10-04T02:00:12.315464+0000 (3600 sec) 2023-10-04T01:00:13.308 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:00:13.310 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2670833392 2023-10-04T01:00:14.117 INFO:tasks.workunit.client.0.smithi027.stderr:listed 23 entries 2023-10-04T01:00:14.142 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:14.142 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:00:20.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.200+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:00:20.209 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:00:20.210 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:00:20.210 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:20.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:00:21.061 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:00:21.062 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:00:21.062 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:00:21.062 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:21.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:00:24.144 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:24.147 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:00:24.147 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:00:24.163 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:00:25.185 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:25.190 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:25.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381225 2023-10-04T01:00:25.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381225 -le 1696383282 ]] 2023-10-04T01:00:25.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:00:25.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:00:25.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:00:35.194 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:00:35.196 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:35.196 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:00:35.200 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:00:36.084 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:36.084 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:36.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381236 2023-10-04T01:00:36.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381236 -le 1696383282 ]] 2023-10-04T01:00:36.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:00:36.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:00:36.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:00:40.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:00:40.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:00:40.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:00:40.592 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:00:40.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:00:40.593 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:40.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:00:46.099 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:46.099 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:00:46.102 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:00:46.109 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:00:46.991 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:46.991 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:46.992 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381246 2023-10-04T01:00:46.992 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381246 -le 1696383282 ]] 2023-10-04T01:00:46.992 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:00:46.992 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:00:46.992 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:00:50.150 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.147+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:00:50.157 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:00:50.157 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:00:50.157 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:00:50.157 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:00:50.157 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:00:50.158 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:00:50.158 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:00:50.158 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:00:50.155+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:00:56.994 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:00:56.996 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:00:56.998 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:00:57.002 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:00:57.853 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:00:57.853 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:00:57.854 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381257 2023-10-04T01:00:57.854 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381257 -le 1696383282 ]] 2023-10-04T01:00:57.854 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:00:57.854 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:00:57.855 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:01:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:00:59.999+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:00:59.999+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b94b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:00:59.999+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:00:59.999+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b94980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.003+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.003+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b94000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.007+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.008 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.007+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b94380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.011+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.012 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.011+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.015+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.015+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.023+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.023+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.023+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.023+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.027+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.027+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.027+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.030 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.031+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.031+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.034 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.031+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b95e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.035+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.035+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.039+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.039+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.043+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.046 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.043+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.051+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.051 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.051+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.055+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.055+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.059+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.060 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.059+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.063+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.063+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.067+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.067+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b22e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.067+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:01:00.067+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b23000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:01:05.953 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:01:05.951+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:01:05.956 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:01:05.951+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:01:07.857 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:01:07.857 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:01:07.859 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:01:07.860 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:01:08.833 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2670833392 2023-10-04T01:01:08.833 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:01:08.833 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381268 2023-10-04T01:01:08.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381268 -le 1696383282 ]] 2023-10-04T01:01:08.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2670833392 ]] 2023-10-04T01:01:08.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2670833392 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:01:08.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:01:18.839 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:01:18.844 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:01:18.852 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:01:18.852 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:01:19.758 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:01:19.759 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:01:19.759 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381279 2023-10-04T01:01:19.760 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381279 -le 1696383282 ]] 2023-10-04T01:01:19.760 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:01:19.760 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\6\7\0\8\3\3\3\9\2 ]] 2023-10-04T01:01:19.760 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1712616634 2023-10-04T01:01:22.142 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1712616634 until 2023-10-04T02:01:21.417713+0000 (3600 sec) 2023-10-04T01:01:22.195 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:01:22.198 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1712616634 2023-10-04T01:01:22.765 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:01:22.758+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:01:22.767 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:01:22.766+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:01:23.023 INFO:tasks.workunit.client.0.smithi027.stderr:listed 24 entries 2023-10-04T01:01:23.084 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:01:23.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:01:28.662 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:01:28.658+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:01:33.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:01:33.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:01:33.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:01:33.104 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:01:33.969 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:01:33.971 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:01:33.976 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381293 2023-10-04T01:01:33.976 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381293 -le 1696383282 ]] 2023-10-04T01:01:33.976 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:01:33.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:01:33.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:01:36.374 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:01:36.374+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:01:43.980 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:01:43.981 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:01:43.981 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:01:43.985 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:01:45.068 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:01:45.069 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:01:45.070 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381305 2023-10-04T01:01:45.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381305 -le 1696383282 ]] 2023-10-04T01:01:45.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:01:45.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:01:45.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:01:55.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:01:55.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:01:55.074 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:01:55.075 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:01:56.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:01:56.017 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:01:56.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381316 2023-10-04T01:01:56.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381316 -le 1696383282 ]] 2023-10-04T01:01:56.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:01:56.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:01:56.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:02:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.001+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.001+0000 7ff17f176640 -1 librbd::ImageState: 0x5644811e7c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.005+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.005+0000 7ff17f176640 -1 librbd::ImageState: 0x564481451500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.009+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.009+0000 7ff17e975640 -1 librbd::ImageState: 0x564481bbe800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.013+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.013+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829af600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.017+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.017+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e6280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.025+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.025+0000 7ff17e975640 -1 librbd::ImageState: 0x564481d8b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.029+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.029+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b22e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.033+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.033+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b23000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.041+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.044 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.041+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b23380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.053+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.053+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b23200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.057+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.056 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.057+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b22b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.061+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.061+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.061+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.061+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258e800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.065+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.065+0000 7ff17e975640 -1 librbd::ImageState: 0x56448239f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.069+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.069+0000 7ff17e975640 -1 librbd::ImageState: 0x564482445f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.073+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.073+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b22e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.073+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.073+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.077+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.077 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.077+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.081+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.081+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b23a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.089+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:00.088 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:00.089+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b23880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:02:01.405 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:02:01.405+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:02:01.405 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:02:01.405+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:02:01.405 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:02:01.405+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:02:06.023 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:02:06.027 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:02:06.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:02:06.029 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:02:07.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:02:07.007 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:02:07.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381327 2023-10-04T01:02:07.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381327 -le 1696383282 ]] 2023-10-04T01:02:07.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:02:07.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:02:07.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:02:17.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:02:17.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:02:17.020 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:02:17.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:02:17.907 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712616634 2023-10-04T01:02:17.908 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:02:17.908 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381337 2023-10-04T01:02:17.908 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381337 -le 1696383282 ]] 2023-10-04T01:02:17.909 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712616634 ]] 2023-10-04T01:02:17.909 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712616634 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:02:17.909 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:02:27.910 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:02:27.911 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:02:27.911 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:02:27.915 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:02:28.763 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:02:28.763 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:02:28.763 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381348 2023-10-04T01:02:28.764 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381348 -le 1696383282 ]] 2023-10-04T01:02:28.764 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:02:28.764 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\6\1\6\6\3\4 ]] 2023-10-04T01:02:28.764 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/503959746 2023-10-04T01:02:31.168 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/503959746 until 2023-10-04T02:02:30.357784+0000 (3600 sec) 2023-10-04T01:02:31.207 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/503959746 2023-10-04T01:02:31.208 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:02:31.894 INFO:tasks.workunit.client.0.smithi027.stderr:listed 25 entries 2023-10-04T01:02:31.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:02:31.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:02:41.730 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:02:41.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:02:41.732 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:02:41.732 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:02:41.728+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:02:41.922 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:02:41.923 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:02:41.932 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:02:41.932 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381362 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381362 -le 1696383282 ]] 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:02:42.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:02:52.891 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:02:52.891 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:02:52.891 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:02:52.900 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:02:53.780 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:02:53.780 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:02:53.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381373 2023-10-04T01:02:53.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381373 -le 1696383282 ]] 2023-10-04T01:02:53.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:02:53.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:02:53.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:59.999+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:02:59.999+0000 7ff17f977640 -1 librbd::ImageState: 0x56448262f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.003+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.003+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.007+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.007+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b22780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.011+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.014 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.011+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b23480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.015+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.019 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.015+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b23680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.023+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.023+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b23b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.029 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.027+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.029 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.027+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b23d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.047+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.047+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b23f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.051+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.051+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfc180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.063+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.064 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.063+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfc880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.068 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.067+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.067+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.071+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.071+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfc500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.075+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.075+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfc980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.083+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.084 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.083+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfcb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.087+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.087+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfcd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.091+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.091+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfcf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.099+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.102 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.099+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfd180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.110 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.107+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.111 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.107+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfd380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.114 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.111+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.114 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.111+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfd580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.124 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.123+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:00.124 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:03:00.123+0000 7ff17f977640 -1 librbd::ImageState: 0x564482bfd780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:03:03.783 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:03:03.784 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:03:03.784 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:03:03.792 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:03:04.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:03:04.748 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:03:04.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381384 2023-10-04T01:03:04.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381384 -le 1696383282 ]] 2023-10-04T01:03:04.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:03:04.749 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:03:04.749 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:10.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:10.703+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:03:10.711 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:10.707+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:03:10.711 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:10.707+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:03:10.711 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:10.707+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:03:13.461 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:13.451+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:03:13.461 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:13.451+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:03:13.461 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:13.451+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:03:13.462 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:13.451+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:03:14.750 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:03:14.752 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:03:14.755 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:03:14.756 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:03:15.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:03:15.660 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:03:15.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381395 2023-10-04T01:03:15.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381395 -le 1696383282 ]] 2023-10-04T01:03:15.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:03:15.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:03:15.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:25.665 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:03:25.665 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:03:25.678 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:03:25.694 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:03:26.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/503959746 2023-10-04T01:03:26.696 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:03:26.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381406 2023-10-04T01:03:26.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381406 -le 1696383282 ]] 2023-10-04T01:03:26.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/503959746 ]] 2023-10-04T01:03:26.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/503959746 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:03:26.698 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:35.984 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:03:35.982+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:03:36.704 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:03:36.705 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:03:36.705 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:03:36.734 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:03:37.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:03:37.563 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:03:37.564 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381417 2023-10-04T01:03:37.564 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381417 -le 1696383282 ]] 2023-10-04T01:03:37.564 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:03:37.564 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\5\0\3\9\5\9\7\4\6 ]] 2023-10-04T01:03:37.564 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1582006596 2023-10-04T01:03:39.369 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1582006596 until 2023-10-04T02:03:38.617207+0000 (3600 sec) 2023-10-04T01:03:39.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1582006596 2023-10-04T01:03:39.405 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:03:40.199 INFO:tasks.workunit.client.0.smithi027.stderr:listed 26 entries 2023-10-04T01:03:40.237 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:03:40.237 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:50.248 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:03:50.248 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:03:50.248 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:03:50.263 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:03:51.218 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:03:51.218 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:03:51.219 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381431 2023-10-04T01:03:51.219 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381431 -le 1696383282 ]] 2023-10-04T01:03:51.219 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:03:51.219 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:03:51.219 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:03:55.697 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.698+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:03:55.712 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:03:55.713 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:03:55.706+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:04:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.001+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.002 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.001+0000 7ff17e975640 -1 librbd::ImageState: 0x5644824e5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.005+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.005+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.017+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.017+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.021+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.021+0000 7ff17e975640 -1 librbd::ImageState: 0x56448258f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.025+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.026 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.025+0000 7ff17e975640 -1 librbd::ImageState: 0x56448239fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.033+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.033+0000 7ff17e975640 -1 librbd::ImageState: 0x56448235b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.037+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.037+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.049+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.049+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.057+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.057+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfcc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.061+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.061+0000 7ff17e975640 -1 librbd::ImageState: 0x5644825d4400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.065+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.069+0000 7ff17f176640 -1 librbd::ImageState: 0x564482444d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.073+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.073 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.073+0000 7ff17f176640 -1 librbd::ImageState: 0x56448262fd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.077+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.079 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.077+0000 7ff17f176640 -1 librbd::ImageState: 0x564482bfdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.081+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.081+0000 7ff17f176640 -1 librbd::ImageState: 0x564482bfda80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.085+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.085+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfd880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.085+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.087 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.085+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfdf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.089 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.089+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.089+0000 7ff17e975640 -1 librbd::ImageState: 0x564482c40100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.093+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.095 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.093+0000 7ff17e975640 -1 librbd::ImageState: 0x564482c40300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.097+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.097+0000 7ff17e975640 -1 librbd::ImageState: 0x564482c40500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.134 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.133+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:00.134 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:04:00.133+0000 7ff17e975640 -1 librbd::ImageState: 0x564482c40700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:04:01.222 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:04:01.227 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:01.229 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:01.230 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:02.187 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:04:02.187 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:04:02.188 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381442 2023-10-04T01:04:02.188 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381442 -le 1696383282 ]] 2023-10-04T01:04:02.188 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:04:02.188 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:04:02.188 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:04:07.027 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:04:07.025+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:04:11.126 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.125+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:04:11.133 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:04:11.134 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:04:11.135 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:11.129+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:04:12.195 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:12.195 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:04:12.195 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:12.213 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:13.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:04:13.035 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:04:13.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381453 2023-10-04T01:04:13.036 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381453 -le 1696383282 ]] 2023-10-04T01:04:13.036 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:04:13.036 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:04:13.036 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:04:23.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:23.039 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:04:23.041 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:23.048 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:23.956 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:04:23.956 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:04:23.956 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381463 2023-10-04T01:04:23.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381463 -le 1696383282 ]] 2023-10-04T01:04:23.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:04:23.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:04:23.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:04:33.959 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:33.960 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:33.964 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:04:33.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:35.043 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1582006596 2023-10-04T01:04:35.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:04:35.046 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381475 2023-10-04T01:04:35.046 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381475 -le 1696383282 ]] 2023-10-04T01:04:35.046 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1582006596 ]] 2023-10-04T01:04:35.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1582006596 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:04:35.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:04:45.054 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:04:45.055 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:45.059 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:45.069 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:46.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:04:46.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:04:46.028 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381486 2023-10-04T01:04:46.029 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381486 -le 1696383282 ]] 2023-10-04T01:04:46.029 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:04:46.029 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\5\8\2\0\0\6\5\9\6 ]] 2023-10-04T01:04:46.029 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3739722863 2023-10-04T01:04:47.300 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:47.300+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:04:47.301 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:04:47.300+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:04:48.396 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3739722863 until 2023-10-04T02:04:47.642667+0000 (3600 sec) 2023-10-04T01:04:48.419 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:04:48.428 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3739722863 2023-10-04T01:04:49.242 INFO:tasks.workunit.client.0.smithi027.stderr:listed 27 entries 2023-10-04T01:04:49.291 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:04:49.292 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:04:59.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:04:59.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:04:59.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:04:59.295 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.000+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.000+0000 7ff17f176640 -1 librbd::ImageState: 0x564482bfde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.004 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.004+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.005 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c40080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.008+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.009 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.008+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c40900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.016+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.016 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.016+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c40b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.024+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.024+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c40d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.028+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.028+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c40f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.032+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.032+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.036+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.036+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.040+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.040+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.048+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.048+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.056+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.056+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.060+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.060+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.064+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.064+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.072+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.072+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.076+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.078 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.076+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.082 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.080+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.080+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.092+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.093 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.092+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.096+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.098 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.096+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.103 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.100+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.103 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:05:00.100+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b78b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:05:00.225 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:00.228 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:00.232 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381500 2023-10-04T01:05:00.232 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381500 -le 1696383282 ]] 2023-10-04T01:05:00.232 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:00.232 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:00.232 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:10.239 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:05:10.240 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:05:10.245 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:10.264 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:05:11.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:11.285 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:11.286 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381511 2023-10-04T01:05:11.286 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381511 -le 1696383282 ]] 2023-10-04T01:05:11.286 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:11.287 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:11.287 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:21.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:05:21.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:05:21.300 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:05:21.308 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:22.345 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:22.345 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:22.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381522 2023-10-04T01:05:22.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381522 -le 1696383282 ]] 2023-10-04T01:05:22.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:22.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:22.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:32.363 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:05:32.363 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:05:32.366 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:05:32.377 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:33.483 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:33.483 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:33.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381533 2023-10-04T01:05:33.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381533 -le 1696383282 ]] 2023-10-04T01:05:33.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:33.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:33.485 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:43.485 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:05:43.486 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:05:43.486 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:05:43.486 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:44.399 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:44.399 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:44.401 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381544 2023-10-04T01:05:44.401 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381544 -le 1696383282 ]] 2023-10-04T01:05:44.401 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:44.401 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:44.401 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:54.407 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:05:54.407 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:05:54.407 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:05:54.407 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:05:55.315 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3739722863 2023-10-04T01:05:55.316 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:05:55.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381555 2023-10-04T01:05:55.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381555 -le 1696383282 ]] 2023-10-04T01:05:55.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3739722863 ]] 2023-10-04T01:05:55.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3739722863 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:05:55.320 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:05:59.279 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:05:59.275+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:05:59.279 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:05:59.275+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:01.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.223+0000 7f29e174e640 -1 reset not still connected to 0x55a64cff5a00 2023-10-04T01:06:01.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.223+0000 7f29e174e640 -1 reset not still connected to 0x55a661ae6000 2023-10-04T01:06:01.958 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:01.958 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:06:01.959 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:06:01.959 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:06:01.959 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:06:01.959 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:01.955+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:06:05.323 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:06:05.323 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:06:05.323 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:06:05.323 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:06:06.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:06.296 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:06:06.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381566 2023-10-04T01:06:06.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381566 -le 1696383282 ]] 2023-10-04T01:06:06.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2949576082 ]] 2023-10-04T01:06:06.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2949576082 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\3\9\7\2\2\8\6\3 ]] 2023-10-04T01:06:06.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/2949576082 2023-10-04T01:06:08.397 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/2949576082 until 2023-10-04T02:06:07.422707+0000 (3600 sec) 2023-10-04T01:06:08.443 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:06:08.443 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/2949576082 2023-10-04T01:06:09.278 INFO:tasks.workunit.client.0.smithi027.stderr:listed 28 entries 2023-10-04T01:06:09.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:09.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:06:13.091 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.090+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:06:13.107 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:06:13.107 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:06:13.108 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:13.102+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:06:19.304 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:06:19.304 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:06:19.306 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:06:19.333 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:06:20.333 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:20.335 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:06:20.336 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381580 2023-10-04T01:06:20.336 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381580 -le 1696383282 ]] 2023-10-04T01:06:20.336 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2949576082 ]] 2023-10-04T01:06:20.336 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2949576082 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\9\4\9\5\7\6\0\8\2 ]] 2023-10-04T01:06:20.337 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:06:27.263 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:27.263 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:06:27.264 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:06:27.264 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:06:27.264 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:06:27.264 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:27.258+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:06:30.339 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:06:30.340 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:06:30.343 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:06:30.343 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:06:31.256 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:31.258 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:06:31.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381591 2023-10-04T01:06:31.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381591 -le 1696383282 ]] 2023-10-04T01:06:31.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2949576082 ]] 2023-10-04T01:06:31.260 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2949576082 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\9\4\9\5\7\6\0\8\2 ]] 2023-10-04T01:06:31.260 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:06:41.261 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:06:41.262 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:06:41.267 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:06:41.284 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:06:42.221 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:42.222 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:06:42.223 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381602 2023-10-04T01:06:42.223 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381602 -le 1696383282 ]] 2023-10-04T01:06:42.223 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2949576082 ]] 2023-10-04T01:06:42.223 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2949576082 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\9\4\9\5\7\6\0\8\2 ]] 2023-10-04T01:06:42.224 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:06:46.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:46.314+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:06:49.520 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.521+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:06:49.528 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:06:49.528 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:06:49.528 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:06:49.528 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:06:49.529 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:06:49.529 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:06:49.529 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:06:49.529 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:06:49.529+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:06:52.226 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:06:52.227 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:06:52.231 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:06:52.239 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:06:53.196 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/2949576082 2023-10-04T01:06:53.196 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:06:53.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381613 2023-10-04T01:06:53.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381613 -le 1696383282 ]] 2023-10-04T01:06:53.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/2949576082 ]] 2023-10-04T01:06:53.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/2949576082 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\9\4\9\5\7\6\0\8\2 ]] 2023-10-04T01:06:53.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:01.596 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:01.597 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:01.597 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:01.602 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.601+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:01.602 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.601+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:01.602 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:01.601+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:02.079 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:02.073+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:02.079 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:02.077+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:02.079 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:02.077+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:02.080 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:02.077+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:02.080 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:02.077+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:03.199 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:07:03.200 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:07:03.208 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:07:03.213 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:07:03.595 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:07:03.589+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:07:03.595 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:07:03.589+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:07:03.595 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:07:03.589+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:07:04.200 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:04.200 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:07:04.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381624 2023-10-04T01:07:04.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381624 -le 1696383282 ]] 2023-10-04T01:07:04.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:07:04.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\2\9\4\9\5\7\6\0\8\2 ]] 2023-10-04T01:07:04.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3447838188 2023-10-04T01:07:04.987 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:07:04.985+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e152e33450 2023-10-04T01:07:05.462 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x564481b1bb00 image watch failed: 94852249769984, (107) Transport endpoint is not connected 2023-10-04T01:07:05.463 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::Watcher: 0x564481b1bb00 handle_error: handle=94852249769984: (107) Transport endpoint is not connected 2023-10-04T01:07:05.463 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x564481b57200 image watch failed: 94852247314432, (107) Transport endpoint is not connected 2023-10-04T01:07:05.463 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::Watcher: 0x564481b57200 handle_error: handle=94852247314432: (107) Transport endpoint is not connected 2023-10-04T01:07:05.463 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x56448272a300 image watch failed: 94852248114176, (107) Transport endpoint is not connected 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.461+0000 7ff17f977640 -1 librbd::Watcher: 0x56448272a300 handle_error: handle=94852248114176: (107) Transport endpoint is not connected 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x5644814a4870 handle_unwatch client blocklisted 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::Watcher: 0x564481b1bb00 handle_rewatch: client blocklisted 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x5644814a4960 handle_unwatch client blocklisted 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::Watcher: 0x564481b57200 handle_rewatch: client blocklisted 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x5644814a54a0 handle_unwatch client blocklisted 2023-10-04T01:07:05.464 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::Watcher: 0x56448272a300 handle_rewatch: client blocklisted 2023-10-04T01:07:05.465 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x56448269c600 image watch failed: 94852248110080, (107) Transport endpoint is not connected 2023-10-04T01:07:05.465 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::Watcher: 0x56448269c600 handle_error: handle=94852248110080: (107) Transport endpoint is not connected 2023-10-04T01:07:05.465 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x5644814a54f0 handle_unwatch client blocklisted 2023-10-04T01:07:05.466 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.465+0000 7ff17f977640 -1 librbd::Watcher: 0x56448269c600 handle_rewatch: client blocklisted 2023-10-04T01:07:05.488 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.489+0000 7ff17f977640 -1 librbd::managed_lock::GetLockerRequest: 0x56448172dab0 handle_get_lockers: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.488 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.489+0000 7ff17f977640 -1 librbd::managed_lock::AcquireRequest: 0x564482e2d0e0 handle_get_locker: failed to retrieve lockers: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.489 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.489+0000 7ff17f977640 -1 librbd::ManagedLock: 0x5644816be838 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.489 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.489+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc0d00 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.489 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.489+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x56447cdfab00 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.495 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.493+0000 7ff17f977640 -1 librbd::managed_lock::BreakRequest: 0x564481756780 handle_get_watchers: failed to retrieve watchers: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.495 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.493+0000 7ff17f977640 -1 librbd::managed_lock::AcquireRequest: 0x564482e2dc20 handle_break_lock: failed to break lock : (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.495 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.493+0000 7ff17f977640 -1 librbd::ManagedLock: 0x5644811fa358 handle_acquire_lock: failed to acquire exclusive lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.495 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.493+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc05b0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.495 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.493+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x564482697e40 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.523 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x564481da0600 image watch failed: 94852249764864, (107) Transport endpoint is not connected 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::Watcher: 0x564481da0600 handle_error: handle=94852249764864: (107) Transport endpoint is not connected 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x564481da1200 image watch failed: 94852253178880, (107) Transport endpoint is not connected 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::Watcher: 0x564481da1200 handle_error: handle=94852253178880: (107) Transport endpoint is not connected 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x564481e14e60 handle_unwatch client blocklisted 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::Watcher: 0x564481da0600 handle_rewatch: client blocklisted 2023-10-04T01:07:05.524 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x564481524eb0 handle_unwatch client blocklisted 2023-10-04T01:07:05.525 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.525+0000 7ff17f977640 -1 librbd::Watcher: 0x564481da1200 handle_rewatch: client blocklisted 2023-10-04T01:07:05.608 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::ImageWatcher: 0x564481721b00 image watch failed: 94852247568384, (107) Transport endpoint is not connected 2023-10-04T01:07:05.608 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::Watcher: 0x564481721b00 handle_error: handle=94852247568384: (107) Transport endpoint is not connected 2023-10-04T01:07:05.610 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x564481d5eaf0 handle_unwatch client blocklisted 2023-10-04T01:07:05.610 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::Watcher: 0x564481721b00 handle_rewatch: client blocklisted 2023-10-04T01:07:05.610 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::ManagedLock: 0x564481ae5bb8 send_acquire_lock: blocklisted waiting for exclusive lock 2023-10-04T01:07:05.610 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::ManagedLock: 0x564481ae5bb8 send_acquire_lock: watcher not registered - client blocklisted 2023-10-04T01:07:05.611 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::mirror::snapshot::UnlinkPeerRequest: 0x564482e47d40 handle_remove_snapshot: failed to remove snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.611 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.609+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1450 handle_unlink_peer: failed to unlink peer: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.616 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.617+0000 7ff17f176640 -1 librbd::ImageWatcher: 0x564482650000 image watch failed: 94852251457536, (107) Transport endpoint is not connected 2023-10-04T01:07:05.616 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.617+0000 7ff17f176640 -1 librbd::Watcher: 0x564482650000 handle_error: handle=94852251457536: (107) Transport endpoint is not connected 2023-10-04T01:07:05.617 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.617+0000 7ff17f977640 -1 librbd::image_watcher::NotifyLockOwner: 0x56448262f780 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.617 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.617+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc0b60 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.617 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.617+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x564482800dc0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:05.625 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:05.625+0000 7ff17f977640 -1 librbd::watcher::RewatchRequest: 0x56448141e320 handle_unwatch client blocklisted 2023-10-04T01:07:06.462 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3447838188 until 2023-10-04T02:07:05.651429+0000 (3600 sec) 2023-10-04T01:07:06.473 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1860 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x564482a66420 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1380 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x5644828e9080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481acc1a0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.474 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x5644828f9ce0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.475 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1110 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.475 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.473+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482555080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.475 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.476 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.476 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482800dc0 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.476 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.476 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482e47dc0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.476 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482800f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482e479c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482b42000 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482999640 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482697080 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.477 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482e462c0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.478 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.478 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.479 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f977640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.479 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f977640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.479 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f977640 -1 librbd::managed_lock::ReleaseRequest: 0x564481ff4960 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.479 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.477+0000 7ff17f176640 -1 librbd::managed_lock::ReleaseRequest: 0x5644823855c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.485 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.481+0000 7ff17f977640 -1 librbd::managed_lock::ReleaseRequest: 0x564482118480 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.485 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.481+0000 7ff17f977640 -1 librbd::managed_lock::ReleaseRequest: 0x564481ff4360 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:07:06.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3447838188 2023-10-04T01:07:06.721 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f977640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.721 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f977640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481acc340 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.721 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x564482555760 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.722 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f176640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.722 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f176640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x564481dc1e10 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.722 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.721+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x5644826d7600 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f176640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f176640 -1 librbd::io::AioCompletion: 0x564482a66420 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f176640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482be6c00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f977640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f977640 -1 librbd::io::AioCompletion: 0x564482800f20 fail: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f977640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x564482e47bc0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.728 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f977640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.729 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.729+0000 7ff17f977640 -1 librbd::managed_lock::ReleaseRequest: 0x564481d1f320 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:06.805 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:06.805+0000 7ff187987640 -1 calc_pg_upmaps abort due to max <= 0 2023-10-04T01:07:07.165 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:07.165+0000 7ff17f176640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:07.416 INFO:tasks.workunit.client.0.smithi027.stderr:listed 29 entries 2023-10-04T01:07:07.439 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:07.439 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:07.687 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:07:07.689+0000 7ff17f176640 -1 librbd::managed_lock::ReleaseRequest: 0x56448161a8a0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:07:12.896 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.889+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:07:12.900 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.901+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:07:12.900 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.901+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:12.900 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.901+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:12.900 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.901+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:12.900 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:12.901+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:17.450 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:07:17.451 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:07:17.451 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:07:17.471 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:07:18.463 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:18.467 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:07:18.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381638 2023-10-04T01:07:18.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381638 -le 1696383282 ]] 2023-10-04T01:07:18.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:07:18.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:07:18.469 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:18.678 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.677+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:07:18.682 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:07:18.682 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:18.682 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:18.683 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:18.819 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.813+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:18.819 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.813+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:18.819 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.813+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:18.825 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.825+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:18.825 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.825+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:18.825 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:18.825+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:20.199 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.197+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:20.199 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.197+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:20.226 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.225+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:20.226 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.225+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:20.226 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.225+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:20.226 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:20.225+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:25.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.341+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:07:25.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.341+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:25.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.341+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:25.344 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.341+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:25.345 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.345+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:25.345 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.345+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:25.345 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.345+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:25.346 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.345+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:25.637 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.637+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:07:25.658 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:07:25.658 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:25.658 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:25.658 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:25.658 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:25.659 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:25.659 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:25.659 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:25.649+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:28.476 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:07:28.477 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:07:28.477 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:07:28.497 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:07:29.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:29.570 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:07:29.576 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381649 2023-10-04T01:07:29.576 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381649 -le 1696383282 ]] 2023-10-04T01:07:29.576 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:07:29.577 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:07:29.577 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:39.581 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:07:39.582 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:07:39.583 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:07:39.587 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:07:40.598 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:40.598 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:07:40.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381660 2023-10-04T01:07:40.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381660 -le 1696383282 ]] 2023-10-04T01:07:40.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:07:40.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:07:40.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:42.063 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:42.064 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:42.064 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.060+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:42.070 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.068+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:42.070 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.068+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:42.070 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.068+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:42.070 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:42.068+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:46.409 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:07:46.404+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:07:46.958 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.956+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:07:46.958 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.956+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:07:46.966 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.964+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:07:46.975 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.972+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:07:46.988 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.984+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:07:46.988 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.984+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:46.988 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.984+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:07:46.988 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:46.984+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:07:50.604 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:07:50.604 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:07:50.607 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:07:50.610 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:07:51.641 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:07:51.641 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:07:51.643 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381671 2023-10-04T01:07:51.643 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381671 -le 1696383282 ]] 2023-10-04T01:07:51.644 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:07:51.644 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:07:51.644 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:07:54.549 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:54.543+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:07:54.550 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:07:54.547+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:08:01.645 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:01.645 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:01.648 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:08:01.649 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:02.568 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3447838188 2023-10-04T01:08:02.568 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:08:02.569 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381682 2023-10-04T01:08:02.569 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381682 -le 1696383282 ]] 2023-10-04T01:08:02.569 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3447838188 ]] 2023-10-04T01:08:02.569 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3447838188 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:08:02.569 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:08:12.577 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:08:12.577 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:12.577 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:12.577 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:13.487 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:08:13.493 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:08:13.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381693 2023-10-04T01:08:13.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381693 -le 1696383282 ]] 2023-10-04T01:08:13.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:08:13.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\4\4\7\8\3\8\1\8\8 ]] 2023-10-04T01:08:13.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/1712573181 2023-10-04T01:08:15.496 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/1712573181 until 2023-10-04T02:08:14.766215+0000 (3600 sec) 2023-10-04T01:08:15.523 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:08:15.523 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/1712573181 2023-10-04T01:08:16.276 INFO:tasks.workunit.client.0.smithi027.stderr:listed 30 entries 2023-10-04T01:08:16.305 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:08:16.305 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:08:26.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:26.315 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:26.315 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:08:26.319 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:27.296 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:08:27.296 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:08:27.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381707 2023-10-04T01:08:27.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381707 -le 1696383282 ]] 2023-10-04T01:08:27.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:08:27.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:08:27.297 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:08:33.806 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:33.802+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:08:33.815 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:33.810+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:08:33.815 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:33.810+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:08:36.295 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:36.290+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:08:36.303 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:36.298+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:08:36.304 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:36.298+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:08:36.304 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:36.298+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:08:36.304 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:08:36.298+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:08:37.312 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:37.312 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:08:37.312 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:37.317 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:38.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:08:38.160 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:08:38.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381718 2023-10-04T01:08:38.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381718 -le 1696383282 ]] 2023-10-04T01:08:38.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:08:38.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:08:38.161 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:08:48.177 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:48.180 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:08:48.181 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:48.184 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:49.059 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:08:49.059 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:08:49.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381729 2023-10-04T01:08:49.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381729 -le 1696383282 ]] 2023-10-04T01:08:49.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:08:49.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:08:49.060 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:08:59.062 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:08:59.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:08:59.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:08:59.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.005+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.005+0000 7ff17e975640 -1 librbd::ImageState: 0x5644813e7100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:09:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.009+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.009+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829af180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.013+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.013+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829af600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.019 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:00.020 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.021+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.021+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aec00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.021+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.021 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.021+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aeb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381740 2023-10-04T01:09:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381740 -le 1696383282 ]] 2023-10-04T01:09:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:09:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:09:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:09:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.025+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.025 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.025+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.025+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.028 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.025+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfcf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.029+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.029+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfdf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.033+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.036 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.033+0000 7ff17f176640 -1 librbd::ImageState: 0x564482bfd080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.037+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.037+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfca00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.041+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.044 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.041+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfcc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.045+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.049+0000 7ff17e975640 -1 librbd::ImageState: 0x5644829aee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.053+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.053+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.057+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.057 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.057+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.061+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.061+0000 7ff17e975640 -1 librbd::ImageState: 0x564482bfde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.065+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.066 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.065+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.069+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.071 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.069+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b78f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.073+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.075 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.073+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b78d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.077+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.080 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.077+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.081+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:00.083 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:09:00.081+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:09:10.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:09:10.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:09:10.039 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:10.045 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:09:10.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/1712573181 2023-10-04T01:09:10.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:10.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381750 2023-10-04T01:09:10.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381750 -le 1696383282 ]] 2023-10-04T01:09:10.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/1712573181 ]] 2023-10-04T01:09:10.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/1712573181 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:09:10.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:09:20.925 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:09:20.925 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:09:20.930 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:20.948 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:09:21.983 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:09:21.984 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:21.984 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381761 2023-10-04T01:09:21.984 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381761 -le 1696383282 ]] 2023-10-04T01:09:21.985 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:09:21.985 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\1\7\1\2\5\7\3\1\8\1 ]] 2023-10-04T01:09:21.985 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3677686417 2023-10-04T01:09:24.497 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3677686417 until 2023-10-04T02:09:23.731817+0000 (3600 sec) 2023-10-04T01:09:24.530 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:09:24.530 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3677686417 2023-10-04T01:09:25.240 INFO:tasks.workunit.client.0.smithi027.stderr:listed 31 entries 2023-10-04T01:09:25.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:09:25.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:09:35.276 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:09:35.276 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:09:35.277 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:09:35.277 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:36.258 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:09:36.258 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:36.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381776 2023-10-04T01:09:36.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381776 -le 1696383282 ]] 2023-10-04T01:09:36.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:09:36.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:09:36.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:09:44.769 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:09:44.764+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:09:44.770 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:09:44.764+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:09:44.770 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:09:44.764+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:09:46.273 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:09:46.273 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:09:46.282 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:46.283 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:09:47.375 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:09:47.375 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:47.375 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381787 2023-10-04T01:09:47.376 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381787 -le 1696383282 ]] 2023-10-04T01:09:47.376 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:09:47.376 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:09:47.376 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:09:57.379 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:09:57.379 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:09:57.385 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:09:57.388 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:09:58.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:09:58.247 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:09:58.248 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381798 2023-10-04T01:09:58.248 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381798 -le 1696383282 ]] 2023-10-04T01:09:58.248 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:09:58.248 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:09:58.248 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f977640 -1 librbd::ImageState: 0x564482c40d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f176640 -1 librbd::ImageState: 0x564481450b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.010 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.007+0000 7ff17f977640 -1 librbd::ImageState: 0x56448235b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.011+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.013 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.011+0000 7ff17f977640 -1 librbd::ImageState: 0x5644824e4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.011+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.015 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.011+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b78f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.019+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.024 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.019+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b78100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.027+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.033 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.031+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b79980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.043 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.039+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.044 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.039+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b79800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.043+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.043+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b79280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.047+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.052 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.047+0000 7ff17f977640 -1 librbd::ImageState: 0x564482b79b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.051+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.055 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.051+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.055+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.058 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.055+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.059+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.061 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.059+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.063+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.063+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.063+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234c680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.069 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.063+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234c880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.071+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.074 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.071+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.079+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.079+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.087+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.087+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.094 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.091+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.094 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.091+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.099 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.095+0000 7ff17f977640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:00.099 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:10:00.095+0000 7ff17f977640 -1 librbd::ImageState: 0x56448234da00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:10:08.254 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:10:08.263 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:10:08.265 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:10:08.273 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:10:09.125 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:10:09.125 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:10:09.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381809 2023-10-04T01:10:09.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381809 -le 1696383282 ]] 2023-10-04T01:10:09.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:10:09.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:10:09.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:10:17.556 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:17.547+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:10:17.556 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:17.551+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:10:19.128 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:10:19.129 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:10:19.129 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:10:19.144 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:10:20.122 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3677686417 2023-10-04T01:10:20.122 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:10:20.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381820 2023-10-04T01:10:20.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381820 -le 1696383282 ]] 2023-10-04T01:10:20.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3677686417 ]] 2023-10-04T01:10:20.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3677686417 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:10:20.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:10:26.991 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:26.986+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:10:26.991 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:26.986+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:10:26.992 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:26.986+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:10:30.126 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:10:30.126 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:10:30.133 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:10:30.149 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:10:31.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:10:31.147 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:10:31.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381831 2023-10-04T01:10:31.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381831 -le 1696383282 ]] 2023-10-04T01:10:31.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:10:31.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\6\7\7\6\8\6\4\1\7 ]] 2023-10-04T01:10:31.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3821044334 2023-10-04T01:10:33.277 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3821044334 until 2023-10-04T02:10:32.454054+0000 (3600 sec) 2023-10-04T01:10:33.321 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:10:33.322 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3821044334 2023-10-04T01:10:34.025 INFO:tasks.workunit.client.0.smithi027.stderr:listed 32 entries 2023-10-04T01:10:34.051 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:10:34.051 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:10:44.044 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.038+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:10:44.050 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:10:44.050 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:10:44.050 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:10:44.051 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:10:44.051 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:10:44.051 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:44.046+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:10:44.065 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:10:44.066 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:10:44.070 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:10:44.077 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:10:45.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:10:45.011 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:10:45.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381845 2023-10-04T01:10:45.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381845 -le 1696383282 ]] 2023-10-04T01:10:45.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:10:45.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:10:45.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:10:50.159 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:10:50.158+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:10:50.160 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:10:50.158+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:10:50.160 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:10:50.158+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:10:54.676 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:10:54.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:10:54.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:10:55.019 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:10:55.020 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:10:55.020 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:10:55.027 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:10:56.149 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:10:56.149 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:10:56.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381856 2023-10-04T01:10:56.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381856 -le 1696383282 ]] 2023-10-04T01:10:56.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:10:56.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:10:56.150 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:11:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.005+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.005+0000 7ff17e975640 -1 librbd::ImageState: 0x56448262f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.005+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.005+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b78e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.009+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.009+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.017+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.017+0000 7ff17e975640 -1 librbd::ImageState: 0x564482b79300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.025+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.027 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.025+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.033+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.038 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.033+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.041+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.041+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234d980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.045+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.048 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.045+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.049+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.049+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.061+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.063 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.061+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.065+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.065 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.065+0000 7ff17e975640 -1 librbd::ImageState: 0x56448234de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.089+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.090 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.089+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.089+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.092 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.089+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.097+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.105 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.097+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.107 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.108 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.105+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.120 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.117+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.120 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.117+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.124 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.121+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.124 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.121+0000 7ff17e975640 -1 librbd::ImageState: 0x564482746e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.127 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.125+0000 7ff17e975640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.127 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:11:00.125+0000 7ff17e975640 -1 librbd::ImageState: 0x564482747080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:11:00.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:00.225+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:11:00.234 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:00.233+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:11:00.234 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:00.233+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:11:00.234 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:00.233+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:11:06.153 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:11:06.153 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:11:06.169 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:11:06.177 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:11:07.068 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:11:07.068 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:11:07.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381867 2023-10-04T01:11:07.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381867 -le 1696383282 ]] 2023-10-04T01:11:07.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:11:07.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:11:07.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:11:11.808 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:11.801+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:11:11.809 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:11.809+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:11:11.809 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:11.809+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:11:11.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:11.813+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:11:11.814 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:11:11.813+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:11:17.071 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:11:17.072 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:11:17.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:11:17.079 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:11:18.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:11:18.014 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:11:18.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381878 2023-10-04T01:11:18.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381878 -le 1696383282 ]] 2023-10-04T01:11:18.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:11:18.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:11:18.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:11:28.021 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:11:28.021 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:11:28.022 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:11:28.022 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:11:29.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3821044334 2023-10-04T01:11:29.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:11:29.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381889 2023-10-04T01:11:29.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381889 -le 1696383282 ]] 2023-10-04T01:11:29.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3821044334 ]] 2023-10-04T01:11:29.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3821044334 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:11:29.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:11:39.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:11:39.029 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:11:39.044 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:11:39.045 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:11:39.952 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:11:39.953 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:11:39.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381899 2023-10-04T01:11:39.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381899 -le 1696383282 ]] 2023-10-04T01:11:39.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:11:39.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\8\2\1\0\4\4\3\3\4 ]] 2023-10-04T01:11:39.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist add 172.21.15.27:0/3787056471 2023-10-04T01:11:42.282 INFO:tasks.workunit.client.0.smithi027.stderr:blocklisting 172.21.15.27:0/3787056471 until 2023-10-04T02:11:41.540101+0000 (3600 sec) 2023-10-04T01:11:42.355 INFO:tasks.workunit.client.0.smithi027.stderr:+ ceph osd blocklist ls 2023-10-04T01:11:42.355 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep -q 172.21.15.27:0/3787056471 2023-10-04T01:11:43.174 INFO:tasks.workunit.client.0.smithi027.stderr:listed 33 entries 2023-10-04T01:11:43.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ PREV_CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:11:43.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:11:53.202 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:11:53.204 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:11:53.205 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:11:53.220 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:11:54.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:11:54.163 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:11:54.163 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381914 2023-10-04T01:11:54.163 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381914 -le 1696383282 ]] 2023-10-04T01:11:54.163 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:11:54.164 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:11:54.164 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.000+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.003 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.000+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b79700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.004+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.007 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.004+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b95f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.008+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.011 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.008+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b23c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.012+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.017 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.012+0000 7ff17f176640 -1 librbd::ImageState: 0x564482b95d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.020+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.023 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.020+0000 7ff17f176640 -1 librbd::ImageState: 0x56448234c080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.028+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.032 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.028+0000 7ff17f176640 -1 librbd::ImageState: 0x564482c41000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.032+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.035 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.032+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.036+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.037 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.036+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.039 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.036+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.040 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.036+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.040+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.042 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.040+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.040+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.045 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.040+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.048+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.050 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.048+0000 7ff17f176640 -1 librbd::ImageState: 0x5644813e7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.052+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.054 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.052+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.056+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.059 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.056+0000 7ff17f176640 -1 librbd::ImageState: 0x564482746e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.060+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.062 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.060+0000 7ff17f176640 -1 librbd::ImageState: 0x564482747080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.064+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.067 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.064+0000 7ff17f176640 -1 librbd::ImageState: 0x564482747280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.068+0000 7ff17f176640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:00.070 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:12:00.068+0000 7ff17f176640 -1 librbd::ImageState: 0x564482747400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-04T01:12:04.166 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:12:04.166 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:04.170 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:04.170 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:05.083 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:12:05.084 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:12:05.095 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381925 2023-10-04T01:12:05.095 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381925 -le 1696383282 ]] 2023-10-04T01:12:05.095 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:12:05.095 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:12:05.095 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:15.115 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:15.117 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:15.117 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:12:15.120 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:16.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:12:16.012 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:12:16.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381936 2023-10-04T01:12:16.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381936 -le 1696383282 ]] 2023-10-04T01:12:16.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:12:16.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:12:16.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:26.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:26.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:26.017 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:26.021 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:12:27.055 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381947 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381947 -le 1696383282 ]] 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:12:27.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:37.058 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:37.059 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:37.076 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:12:37.080 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:37.165 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:37.159+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:12:37.178 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:37.175+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:12:37.178 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:37.175+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:12:37.178 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:37.175+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:12:37.178 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:37.175+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:12:38.215 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:12:38.215 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:12:38.216 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381958 2023-10-04T01:12:38.216 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381958 -le 1696383282 ]] 2023-10-04T01:12:38.216 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:12:38.216 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:12:38.216 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:48.226 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:48.231 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:12:48.233 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:48.239 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:49.213 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:12:49.213 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:12:49.214 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381969 2023-10-04T01:12:49.214 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381969 -le 1696383282 ]] 2023-10-04T01:12:49.214 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:12:49.214 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:12:49.214 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:12:52.234 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:52.230+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:12:52.235 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:52.230+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:12:54.020 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:12:54.021 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:12:54.010+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:12:59.217 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:12:59.217 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:12:59.217 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:12:59.232 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:00.136 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:00.136 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381980 2023-10-04T01:13:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381980 -le 1696383282 ]] 2023-10-04T01:13:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:13:00.780 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:13:00.781 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:13:00.781 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:00.774+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:13:07.372 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:07.366+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:13:07.373 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:07.366+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:13:10.139 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:13:10.139 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:13:10.139 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:13:10.144 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:11.001 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:11.001 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:11.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696381991 2023-10-04T01:13:11.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696381991 -le 1696383282 ]] 2023-10-04T01:13:11.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:11.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:11.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:21.009 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:13:21.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:13:21.018 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:13:21.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:21.755 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:21.755 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382001 2023-10-04T01:13:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382001 -le 1696383282 ]] 2023-10-04T01:13:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:21.756 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:25.621 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:25.617+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:13:25.624 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:25.621+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:13:25.625 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:25.621+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:13:25.625 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:13:25.621+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:13:31.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:13:31.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:13:31.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:31.765 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:13:32.577 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:32.583 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:32.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382012 2023-10-04T01:13:32.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382012 -le 1696383282 ]] 2023-10-04T01:13:32.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:32.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:32.585 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:41.085 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:13:41.081+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:13:41.086 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:13:41.081+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:13:41.086 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:13:41.081+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:13:42.585 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:13:42.586 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:42.586 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:13:42.587 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:13:43.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:43.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382023 2023-10-04T01:13:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382023 -le 1696383282 ]] 2023-10-04T01:13:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:43.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:53.557 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:13:53.558 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:13:53.561 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:13:53.566 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:13:54.534 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:13:54.541 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:13:54.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382034 2023-10-04T01:13:54.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382034 -le 1696383282 ]] 2023-10-04T01:13:54.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:13:54.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:13:54.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:13:59.889 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:13:59.885+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:14:04.550 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:04.550 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:04.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:14:04.556 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:05.427 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:14:05.428 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:14:05.428 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382045 2023-10-04T01:14:05.428 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382045 -le 1696383282 ]] 2023-10-04T01:14:05.429 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:14:05.429 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:14:05.429 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:14:11.411 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:14:11.408+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:14:11.411 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:14:11.408+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:14:11.412 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:14:11.408+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:14:12.451 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:14:12.448+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:14:12.451 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:14:12.448+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:14:12.451 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:14:12.448+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:14:12.451 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:14:12.448+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:14:12.451 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:14:12.448+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:14:15.430 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:14:15.430 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:15.435 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:15.457 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:16.345 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382056 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382056 -le 1696383282 ]] 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:14:16.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:14:26.353 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:14:26.356 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:26.365 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:26.366 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:27.227 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:14:27.227 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:14:27.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382067 2023-10-04T01:14:27.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382067 -le 1696383282 ]] 2023-10-04T01:14:27.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:14:27.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:14:27.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:14:37.231 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:37.232 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:14:37.236 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:37.236 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:38.081 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:14:38.081 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:14:38.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382078 2023-10-04T01:14:38.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382078 -le 1696383282 ]] 2023-10-04T01:14:38.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:14:38.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:14:38.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:14:48.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:14:48.088 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:48.097 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:48.113 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:49.126 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:14:49.126 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:14:49.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382089 2023-10-04T01:14:49.127 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382089 -le 1696383282 ]] 2023-10-04T01:14:49.128 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:14:49.128 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:14:49.128 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:14:59.129 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:14:59.131 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:14:59.133 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:14:59.150 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:00.168 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:00.170 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:00.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382100 2023-10-04T01:15:00.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382100 -le 1696383282 ]] 2023-10-04T01:15:00.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:00.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:00.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:15:10.174 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:10.174 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:15:10.174 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:15:10.190 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:15:11.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:11.017 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:11.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382111 2023-10-04T01:15:11.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382111 -le 1696383282 ]] 2023-10-04T01:15:11.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:11.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:11.019 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:15:21.021 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:15:21.023 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:21.023 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:15:21.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:15:21.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:21.958 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:21.958 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382121 2023-10-04T01:15:21.958 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382121 -le 1696383282 ]] 2023-10-04T01:15:21.959 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:21.959 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:21.959 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:15:31.970 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:15:31.973 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:15:31.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:15:31.986 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:32.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:32.927 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:32.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382132 2023-10-04T01:15:32.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382132 -le 1696383282 ]] 2023-10-04T01:15:32.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:32.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:32.928 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:15:42.934 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:42.934 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:15:42.934 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:15:42.936 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:15:43.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:43.834 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:43.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382143 2023-10-04T01:15:43.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382143 -le 1696383282 ]] 2023-10-04T01:15:43.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:43.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:43.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:15:50.578 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:15:50.574+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:15:53.836 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:15:53.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:15:53.840 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:15:53.855 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:15:54.829 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:15:54.833 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:15:54.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382154 2023-10-04T01:15:54.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382154 -le 1696383282 ]] 2023-10-04T01:15:54.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:15:54.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:15:54.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:04.840 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:04.847 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:04.852 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:04.856 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:05.815 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:16:05.816 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:16:05.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382165 2023-10-04T01:16:05.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382165 -le 1696383282 ]] 2023-10-04T01:16:05.817 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:16:05.817 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:16:05.817 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:10.521 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:10.518+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:16:10.531 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:10.530+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:16:10.532 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:10.530+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:16:10.532 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:10.530+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:16:11.927 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:11.926+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:16:12.646 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:12.641+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:16:12.646 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:12.641+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:16:12.646 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:12.641+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:16:15.819 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:15.821 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:15.828 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:15.833 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:16.698 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:16:16.703 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:16:16.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382176 2023-10-04T01:16:16.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382176 -le 1696383282 ]] 2023-10-04T01:16:16.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:16:16.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:16:16.704 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:18.206 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:18.205+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:16:18.211 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:18.209+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:16:18.211 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:18.209+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:16:26.712 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:26.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:26.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:26.721 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:27.638 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382187 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382187 -le 1696383282 ]] 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:16:27.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:32.500 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.497+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:16:32.516 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:16:32.516 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.513+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:16:32.763 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.753+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:16:32.765 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:16:32.766 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:16:32.766 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:16:32.766 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.761+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:16:32.770 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.769+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:16:32.770 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.769+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:16:32.770 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.769+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:16:32.771 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.769+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:16:32.771 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:32.769+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:16:34.585 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.581+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:16:34.597 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:16:34.598 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:16:34.597+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:16:37.665 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:37.665 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:37.665 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:37.669 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:38.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:16:38.633 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:16:38.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382198 2023-10-04T01:16:38.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382198 -le 1696383282 ]] 2023-10-04T01:16:38.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:16:38.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:16:38.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:48.635 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:48.645 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:48.646 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:48.661 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:49.600 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:16:49.601 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:16:49.601 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382209 2023-10-04T01:16:49.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382209 -le 1696383282 ]] 2023-10-04T01:16:49.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:16:49.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:16:49.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:16:58.478 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:58.476+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:16:58.478 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:16:58.476+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:16:59.609 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:16:59.610 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:16:59.611 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:16:59.613 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:16:59.668 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T01:16:59.664+0000 7fe2d7cf4640 -1 reset not still connected to 0x563007f39c70 2023-10-04T01:16:59.668 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T01:16:59.664+0000 7fe2d7cf4640 -1 reset not still connected to 0x563007f39d40 2023-10-04T01:16:59.671 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T01:16:59.668+0000 7fe2d7cf4640 -1 reset not still connected to 0x56300975e1a0 2023-10-04T01:16:59.671 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T01:16:59.668+0000 7fe2d7cf4640 -1 reset not still connected to 0x5630099a31e0 2023-10-04T01:16:59.671 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T01:16:59.668+0000 7fe2d7cf4640 -1 reset not still connected to 0x56300a0d4f70 2023-10-04T01:17:00.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:00.556 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:00.556 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382220 2023-10-04T01:17:00.556 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382220 -le 1696383282 ]] 2023-10-04T01:17:00.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:00.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:00.557 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:10.560 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:17:10.560 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:17:10.566 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:17:10.569 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:17:11.679 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382231 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382231 -le 1696383282 ]] 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:11.680 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:12.588 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.580+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:17:12.589 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.584+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:17:12.589 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.584+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:17:12.589 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.588+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:17:12.599 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.592+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:17:12.599 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.592+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:17:12.599 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.592+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:17:12.599 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.592+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:17:12.599 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:12.592+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:17:18.799 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:17:18.799 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:17:18.799 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:17:18.800 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:17:18.800 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:17:18.800 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:18.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:17:21.681 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:17:21.683 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:17:21.688 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:17:21.697 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:17:22.109 INFO:tasks.workunit.client.0.smithi027.stderr:2023-10-04T01:17:22.108+0000 7fab6dd9b640 0 --2- 172.21.15.27:0/2400426068 >> v2:172.21.15.27:3300/0 conn(0x7fab7006e8e0 0x7fab70071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-04T01:17:22.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:22.876 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:22.876 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382242 2023-10-04T01:17:22.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382242 -le 1696383282 ]] 2023-10-04T01:17:22.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:22.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:22.877 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:31.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:17:31.222 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:17:31.222 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.208+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:17:31.223 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.212+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:17:31.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.224+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:17:31.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.224+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:17:31.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.224+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:17:31.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.224+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:17:31.227 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:31.224+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:17:32.878 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:17:32.878 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:17:32.886 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:17:32.898 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:17:33.751 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:33.751 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:33.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382253 2023-10-04T01:17:33.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382253 -le 1696383282 ]] 2023-10-04T01:17:33.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:33.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:33.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:37.387 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:17:37.383+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:17:37.387 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:17:37.383+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:17:43.754 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:17:43.755 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:17:43.772 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:17:43.773 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:17:44.919 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:44.922 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:44.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382264 2023-10-04T01:17:44.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382264 -le 1696383282 ]] 2023-10-04T01:17:44.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:44.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:44.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:48.074 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:17:48.071+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:17:54.934 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:17:54.935 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:17:54.935 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:17:54.951 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:17:55.849 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:17:55.850 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:17:55.850 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382275 2023-10-04T01:17:55.851 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382275 -le 1696383282 ]] 2023-10-04T01:17:55.851 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:17:55.851 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:17:55.851 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:17:56.595 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:56.591+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:17:56.609 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:17:56.599+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:18:05.853 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:18:05.853 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:18:05.863 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:18:05.870 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:18:06.786 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:18:06.805 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:18:06.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382286 2023-10-04T01:18:06.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382286 -le 1696383282 ]] 2023-10-04T01:18:06.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:18:06.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:18:06.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:18:16.814 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:18:16.814 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:18:16.815 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:18:16.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:18:17.675 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:18:17.676 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:18:17.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382297 2023-10-04T01:18:17.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382297 -le 1696383282 ]] 2023-10-04T01:18:17.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:18:17.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:18:17.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:18:27.679 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:18:27.679 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:18:27.680 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:18:27.699 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:18:28.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:18:28.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:18:28.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382308 2023-10-04T01:18:28.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382308 -le 1696383282 ]] 2023-10-04T01:18:28.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:18:28.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:18:28.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:18:38.554 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:18:38.557 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:18:38.557 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:18:38.557 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:18:39.546 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:18:39.546 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:18:39.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382319 2023-10-04T01:18:39.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382319 -le 1696383282 ]] 2023-10-04T01:18:39.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:18:39.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:18:39.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:18:49.549 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:18:49.549 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:18:49.549 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:18:49.556 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:18:50.535 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:18:50.535 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:18:50.536 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382330 2023-10-04T01:18:50.536 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382330 -le 1696383282 ]] 2023-10-04T01:18:50.536 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:18:50.536 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:18:50.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:18:56.639 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.637+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:18:56.639 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.637+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:18:56.639 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.637+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:18:56.643 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.637+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:18:56.644 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.641+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:18:56.644 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:18:56.641+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:19:00.538 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:00.539 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:00.539 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:00.549 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:01.539 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:01.539 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:01.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382341 2023-10-04T01:19:01.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382341 -le 1696383282 ]] 2023-10-04T01:19:01.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:01.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:01.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:19:11.542 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:11.542 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:11.542 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:11.544 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:12.425 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:12.425 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:12.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382352 2023-10-04T01:19:12.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382352 -le 1696383282 ]] 2023-10-04T01:19:12.426 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:12.427 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:12.427 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:19:17.449 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:19:17.445+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:19:17.449 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:19:17.445+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:19:22.428 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:22.433 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:22.433 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:22.443 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:23.301 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:23.301 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:23.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382363 2023-10-04T01:19:23.303 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382363 -le 1696383282 ]] 2023-10-04T01:19:23.303 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:23.303 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:23.303 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:19:33.304 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:33.305 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:33.305 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:33.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:34.164 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:34.165 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:34.165 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382374 2023-10-04T01:19:34.165 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382374 -le 1696383282 ]] 2023-10-04T01:19:34.165 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:34.166 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:34.166 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:19:41.416 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:19:41.404+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:19:41.417 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:19:41.404+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:19:44.167 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:44.169 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:44.185 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:44.201 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:45.085 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:45.085 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:45.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382385 2023-10-04T01:19:45.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382385 -le 1696383282 ]] 2023-10-04T01:19:45.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:45.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:45.086 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:19:55.092 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:19:55.092 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:19:55.093 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:19:55.093 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:19:56.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:19:56.010 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:19:56.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382396 2023-10-04T01:19:56.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382396 -le 1696383282 ]] 2023-10-04T01:19:56.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:19:56.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:19:56.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:06.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:20:06.025 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:20:06.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:20:06.045 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:20:06.931 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:20:06.931 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:20:06.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382406 2023-10-04T01:20:06.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382406 -le 1696383282 ]] 2023-10-04T01:20:06.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:20:06.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:20:06.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:16.933 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:20:16.933 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:20:16.937 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:20:16.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:20:17.890 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:20:17.891 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:20:17.891 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382417 2023-10-04T01:20:17.891 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382417 -le 1696383282 ]] 2023-10-04T01:20:17.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:20:17.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:20:17.892 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:27.893 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:20:27.893 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:20:27.894 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:20:27.923 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:20:28.910 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:20:28.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:20:28.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382428 2023-10-04T01:20:28.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382428 -le 1696383282 ]] 2023-10-04T01:20:28.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:20:28.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:20:28.923 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:38.926 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:20:38.926 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:20:38.939 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:20:38.952 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:20:39.873 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:20:39.873 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:20:39.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382439 2023-10-04T01:20:39.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382439 -le 1696383282 ]] 2023-10-04T01:20:39.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:20:39.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:20:39.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:49.881 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:20:49.882 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:20:49.898 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:20:49.900 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:20:50.918 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:20:50.918 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:20:50.919 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382450 2023-10-04T01:20:50.919 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382450 -le 1696383282 ]] 2023-10-04T01:20:50.919 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:20:50.919 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:20:50.920 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:20:53.463 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:20:53.458+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:20:53.463 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:20:53.458+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:20:53.463 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:20:53.458+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:20:57.298 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:20:57.290+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:20:57.299 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:20:57.294+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:21:00.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:00.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:00.922 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:00.923 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:01.874 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:01.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:01.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382461 2023-10-04T01:21:01.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382461 -le 1696383282 ]] 2023-10-04T01:21:01.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:01.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:01.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:21:05.161 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:05.154+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:21:05.169 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:05.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:21:05.169 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:05.162+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:21:11.154 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:21:11.150+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:21:11.154 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:21:11.150+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:21:11.879 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:11.879 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:11.885 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:11.906 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:12.935 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:12.946 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:12.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382472 2023-10-04T01:21:12.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382472 -le 1696383282 ]] 2023-10-04T01:21:12.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:12.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:12.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:21:22.962 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:22.964 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:22.968 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:22.979 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:23.921 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:23.921 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:23.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382483 2023-10-04T01:21:23.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382483 -le 1696383282 ]] 2023-10-04T01:21:23.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:23.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:23.922 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:21:31.313 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.305+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:21:31.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:21:31.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:21:31.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:21:31.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:21:31.315 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:21:31.316 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:31.313+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:21:33.924 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:33.924 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:33.924 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:33.924 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:35.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:35.015 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:35.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382495 2023-10-04T01:21:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382495 -le 1696383282 ]] 2023-10-04T01:21:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:21:45.017 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:45.022 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:45.025 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:45.038 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:45.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:45.885 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:45.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382505 2023-10-04T01:21:45.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382505 -le 1696383282 ]] 2023-10-04T01:21:45.887 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:45.887 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:45.887 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:21:48.462 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:48.457+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:21:48.486 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:48.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:21:48.487 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:48.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:21:48.487 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:48.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:21:48.487 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:21:48.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:21:55.890 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:21:55.890 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:21:55.895 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:21:55.909 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:21:56.815 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:21:56.815 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:21:56.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382516 2023-10-04T01:21:56.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382516 -le 1696383282 ]] 2023-10-04T01:21:56.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:21:56.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:21:56.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:06.818 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:22:06.818 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:22:06.821 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:22:06.827 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:22:07.721 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:22:07.721 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:22:07.722 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382527 2023-10-04T01:22:07.722 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382527 -le 1696383282 ]] 2023-10-04T01:22:07.722 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:22:07.722 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:22:07.722 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:17.730 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:22:17.731 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:22:17.731 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:22:17.732 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:22:18.729 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:22:18.729 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:22:18.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382538 2023-10-04T01:22:18.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382538 -le 1696383282 ]] 2023-10-04T01:22:18.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:22:18.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:22:18.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:28.738 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:22:28.748 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:22:28.748 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:22:28.749 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:22:29.417 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:22:29.416+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:22:29.423 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:22:29.420+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:22:29.700 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:22:29.700 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:22:29.701 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382549 2023-10-04T01:22:29.701 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382549 -le 1696383282 ]] 2023-10-04T01:22:29.701 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:22:29.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:22:29.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:39.703 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:22:39.704 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:22:39.704 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:22:39.705 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:22:40.647 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:22:40.647 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:22:40.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382560 2023-10-04T01:22:40.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382560 -le 1696383282 ]] 2023-10-04T01:22:40.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:22:40.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:22:40.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:47.296 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.291+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:22:47.302 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.299+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:22:47.302 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.299+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:22:47.302 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.299+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:22:47.304 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.299+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:22:47.318 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.315+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:22:47.319 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.315+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:22:47.319 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.315+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:22:47.319 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.315+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:22:47.319 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:47.315+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:22:50.656 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:22:50.657 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:22:50.667 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:22:50.689 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:22:51.580 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:22:51.581 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:22:51.581 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382571 2023-10-04T01:22:51.581 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382571 -le 1696383282 ]] 2023-10-04T01:22:51.581 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:22:51.582 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:22:51.582 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:22:55.849 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.843+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:22:55.863 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:55.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:22:58.839 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:58.835+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:22:58.839 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:58.835+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:22:58.844 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:58.839+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:22:58.844 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:22:58.839+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:23:01.583 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:01.585 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:01.592 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:01.602 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:02.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:02.501 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:02.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382582 2023-10-04T01:23:02.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382582 -le 1696383282 ]] 2023-10-04T01:23:02.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:02.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:02.502 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:23:12.503 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:12.509 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:12.512 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:12.513 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:13.454 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:13.454 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:13.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382593 2023-10-04T01:23:13.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382593 -le 1696383282 ]] 2023-10-04T01:23:13.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:13.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:13.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:23:20.156 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:23:20.150+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:23:20.156 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:23:20.150+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:23:23.457 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:23.461 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:23.461 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:23.462 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:24.253 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:24.253 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:24.254 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382604 2023-10-04T01:23:24.254 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382604 -le 1696383282 ]] 2023-10-04T01:23:24.254 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:24.254 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:24.255 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:23:34.258 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:34.258 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:34.261 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:34.267 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:35.287 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:35.287 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:35.288 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382615 2023-10-04T01:23:35.288 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382615 -le 1696383282 ]] 2023-10-04T01:23:35.288 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:35.288 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:35.289 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:23:45.297 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:45.297 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:45.299 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:45.304 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:46.245 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:46.245 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:46.245 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382626 2023-10-04T01:23:46.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382626 -le 1696383282 ]] 2023-10-04T01:23:46.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:46.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:46.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:23:56.257 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:23:56.257 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:23:56.259 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:23:56.280 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:23:57.171 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:23:57.171 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:23:57.172 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382637 2023-10-04T01:23:57.172 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382637 -le 1696383282 ]] 2023-10-04T01:23:57.172 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:23:57.172 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:23:57.172 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:24:07.174 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:24:07.177 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:24:07.182 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:24:07.192 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:24:08.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:24:08.057 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:24:08.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382648 2023-10-04T01:24:08.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382648 -le 1696383282 ]] 2023-10-04T01:24:08.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:24:08.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:24:08.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:24:18.061 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:24:18.061 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:24:18.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:24:18.076 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382659 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382659 -le 1696383282 ]] 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:24:19.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:24:19.035 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:24:29.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:24:29.038 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:24:29.054 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:24:29.058 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:24:29.976 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:24:29.979 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:24:29.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382669 2023-10-04T01:24:29.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382669 -le 1696383282 ]] 2023-10-04T01:24:29.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:24:29.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:24:29.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:24:39.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:24:39.986 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:24:39.997 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:24:40.011 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:24:40.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:24:40.871 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:24:40.874 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382680 2023-10-04T01:24:40.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382680 -le 1696383282 ]] 2023-10-04T01:24:40.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:24:40.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:24:40.875 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:24:50.882 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:24:50.882 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:24:50.887 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:24:50.888 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:24:51.840 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:24:51.841 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:24:51.843 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382691 2023-10-04T01:24:51.843 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382691 -le 1696383282 ]] 2023-10-04T01:24:51.843 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:24:51.843 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:24:51.843 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:01.844 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:01.844 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:01.847 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:01.848 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:02.774 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:02.774 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:02.775 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382702 2023-10-04T01:25:02.775 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382702 -le 1696383282 ]] 2023-10-04T01:25:02.775 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:02.775 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:02.775 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:06.862 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:25:06.859+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:25:12.777 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:12.777 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:12.779 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:12.784 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:13.812 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:13.825 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:13.826 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382713 2023-10-04T01:25:13.826 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382713 -le 1696383282 ]] 2023-10-04T01:25:13.827 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:13.827 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:13.827 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:16.792 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:25:16.787+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:25:16.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:25:16.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:25:16.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:25:16.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:25:23.829 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:23.829 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:23.830 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:23.837 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:24.808 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:24.808 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:24.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382724 2023-10-04T01:25:24.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382724 -le 1696383282 ]] 2023-10-04T01:25:24.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:24.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:24.809 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:34.810 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:34.810 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:34.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:34.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:35.746 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:35.746 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:35.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382735 2023-10-04T01:25:35.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382735 -le 1696383282 ]] 2023-10-04T01:25:35.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:35.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:35.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:43.282 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:25:43.270+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:25:45.752 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:45.752 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:45.753 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:45.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:46.753 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:46.753 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:46.754 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382746 2023-10-04T01:25:46.754 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382746 -le 1696383282 ]] 2023-10-04T01:25:46.754 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:46.754 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:46.755 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:25:56.756 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:25:56.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:25:56.758 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:25:56.760 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:25:57.683 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:25:57.684 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:25:57.684 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382757 2023-10-04T01:25:57.685 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382757 -le 1696383282 ]] 2023-10-04T01:25:57.685 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:25:57.685 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:25:57.685 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:26:07.686 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:26:07.690 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:26:07.692 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:26:07.700 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:26:08.582 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:26:08.582 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:26:08.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382768 2023-10-04T01:26:08.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382768 -le 1696383282 ]] 2023-10-04T01:26:08.583 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:26:08.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:26:08.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:26:13.914 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:26:13.906+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:26:13.920 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:26:13.914+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:26:18.586 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:26:18.586 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:26:18.589 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:26:18.596 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:26:19.500 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:26:19.506 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:26:19.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382779 2023-10-04T01:26:19.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382779 -le 1696383282 ]] 2023-10-04T01:26:19.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:26:19.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:26:19.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:26:20.289 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:26:20.285+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:26:20.289 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:26:20.285+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:26:20.290 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:26:20.285+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:26:29.511 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:26:29.512 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:26:29.512 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:26:29.513 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:26:30.385 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:26:30.385 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:26:30.386 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382790 2023-10-04T01:26:30.387 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382790 -le 1696383282 ]] 2023-10-04T01:26:30.387 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:26:30.387 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:26:30.387 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:26:40.396 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:26:40.397 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:26:40.397 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:26:40.401 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:26:41.438 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:26:41.444 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:26:41.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382801 2023-10-04T01:26:41.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382801 -le 1696383282 ]] 2023-10-04T01:26:41.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:26:41.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:26:41.445 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:26:51.452 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:26:51.458 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:26:51.459 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:26:51.462 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:26:52.283 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:26:52.284 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:26:52.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382812 2023-10-04T01:26:52.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382812 -le 1696383282 ]] 2023-10-04T01:26:52.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:26:52.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:26:52.285 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:02.291 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:02.291 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:02.294 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:02.306 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:03.310 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:03.311 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:03.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382823 2023-10-04T01:27:03.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382823 -le 1696383282 ]] 2023-10-04T01:27:03.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:03.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:03.312 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:13.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:13.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:13.315 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:13.316 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:13.496 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:27:13.492+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:27:14.228 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:14.229 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:14.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382834 2023-10-04T01:27:14.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382834 -le 1696383282 ]] 2023-10-04T01:27:14.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:14.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:14.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:16.943 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:27:16.940+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:27:24.232 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:24.233 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:24.242 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:24.250 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:25.311 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:25.361 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:25.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382845 2023-10-04T01:27:25.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382845 -le 1696383282 ]] 2023-10-04T01:27:25.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:25.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:25.363 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:35.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:35.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:35.314 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:35.322 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:36.173 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:36.173 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:36.173 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382856 2023-10-04T01:27:36.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382856 -le 1696383282 ]] 2023-10-04T01:27:36.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:36.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:36.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:46.176 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:46.176 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:46.191 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:46.193 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:47.189 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:47.191 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:47.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382867 2023-10-04T01:27:47.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382867 -le 1696383282 ]] 2023-10-04T01:27:47.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:47.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:47.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:27:51.280 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:27:51.271+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:27:57.201 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:27:57.202 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:27:57.209 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:27:57.214 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:27:58.130 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:27:58.130 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:27:58.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382878 2023-10-04T01:27:58.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382878 -le 1696383282 ]] 2023-10-04T01:27:58.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:27:58.132 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:27:58.132 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:28:08.133 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:28:08.133 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:28:08.134 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:28:08.146 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:28:09.055 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:28:09.056 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:28:09.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382889 2023-10-04T01:28:09.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382889 -le 1696383282 ]] 2023-10-04T01:28:09.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:28:09.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:28:09.057 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:28:19.059 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:28:19.060 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:28:19.075 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:28:19.078 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:28:19.593 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:28:19.586+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:28:19.594 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:28:19.586+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:28:19.954 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:28:19.954 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:28:19.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382899 2023-10-04T01:28:19.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382899 -le 1696383282 ]] 2023-10-04T01:28:19.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:28:19.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:28:19.955 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:28:23.699 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.690+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:28:23.702 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.694+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:28:23.702 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.694+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:28:23.703 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.694+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:28:23.705 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.698+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:28:23.705 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:28:23.698+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:28:29.976 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:28:29.977 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:28:29.980 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:28:29.985 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:28:31.037 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:28:31.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:28:31.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382911 2023-10-04T01:28:31.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382911 -le 1696383282 ]] 2023-10-04T01:28:31.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:28:31.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:28:31.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:28:41.055 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:28:41.056 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:28:41.062 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:28:41.064 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:28:42.040 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:28:42.040 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:28:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382922 2023-10-04T01:28:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382922 -le 1696383282 ]] 2023-10-04T01:28:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:28:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:28:42.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:28:52.049 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:28:52.049 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:28:52.050 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:28:52.071 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:28:53.002 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:28:53.002 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:28:53.004 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382933 2023-10-04T01:28:53.004 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382933 -le 1696383282 ]] 2023-10-04T01:28:53.004 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:28:53.004 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:28:53.004 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:03.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:03.013 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:03.014 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:03.016 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:04.059 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:04.071 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:04.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382944 2023-10-04T01:29:04.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382944 -le 1696383282 ]] 2023-10-04T01:29:04.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:04.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:04.075 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:14.079 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:14.079 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:14.079 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:14.098 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:14.712 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:29:14.709+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:29:14.712 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:29:14.709+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:29:15.009 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:15.009 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:15.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382955 2023-10-04T01:29:15.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382955 -le 1696383282 ]] 2023-10-04T01:29:15.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:15.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:15.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:25.027 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:25.027 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:25.034 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:25.040 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:26.025 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:26.026 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:26.026 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382966 2023-10-04T01:29:26.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382966 -le 1696383282 ]] 2023-10-04T01:29:26.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:26.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:26.027 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:36.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:36.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:36.037 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:36.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:37.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:37.009 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:37.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382977 2023-10-04T01:29:37.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382977 -le 1696383282 ]] 2023-10-04T01:29:37.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:37.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:37.010 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:40.306 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:29:40.300+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:29:40.306 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:29:40.300+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:29:47.011 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:47.012 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:47.012 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:47.012 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:47.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:47.886 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:47.887 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382987 2023-10-04T01:29:47.887 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382987 -le 1696383282 ]] 2023-10-04T01:29:47.888 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:47.888 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:47.888 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:29:57.889 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:29:57.889 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:29:57.890 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:29:57.914 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:29:58.796 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:29:58.801 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:29:58.802 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696382998 2023-10-04T01:29:58.802 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696382998 -le 1696383282 ]] 2023-10-04T01:29:58.802 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:29:58.802 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:29:58.803 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:30:08.808 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:30:08.808 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:30:08.808 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:30:08.811 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:30:09.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:30:09.716 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:30:09.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383009 2023-10-04T01:30:09.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383009 -le 1696383282 ]] 2023-10-04T01:30:09.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:30:09.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:30:09.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:30:19.718 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:30:19.729 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:30:19.729 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:30:19.729 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:30:20.729 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:30:20.732 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:30:20.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383020 2023-10-04T01:30:20.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383020 -le 1696383282 ]] 2023-10-04T01:30:20.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:30:20.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:30:20.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:30:30.743 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:30:30.744 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:30:30.745 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:30:30.747 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:30:31.710 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:30:31.710 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:30:31.710 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383031 2023-10-04T01:30:31.710 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383031 -le 1696383282 ]] 2023-10-04T01:30:31.710 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:30:31.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:30:31.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:30:31.813 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:30:31.814 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:30:31.806+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:30:41.712 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:30:41.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:30:41.713 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:30:41.714 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:30:42.648 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:30:42.650 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:30:42.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383042 2023-10-04T01:30:42.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383042 -le 1696383282 ]] 2023-10-04T01:30:42.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:30:42.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:30:42.651 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:30:52.686 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:30:52.687 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:30:52.687 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:30:52.711 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:30:53.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:30:53.687 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:30:53.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383053 2023-10-04T01:30:53.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383053 -le 1696383282 ]] 2023-10-04T01:30:53.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:30:53.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:30:53.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:31:01.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:31:01.402+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:31:01.569 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:01.562+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:31:03.690 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:03.690 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:03.691 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:03.697 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:04.608 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:04.608 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:04.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383064 2023-10-04T01:31:04.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383064 -le 1696383282 ]] 2023-10-04T01:31:04.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:04.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:04.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:14.611 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:14.611 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:14.615 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:14.616 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:15.523 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:15.523 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:15.524 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383075 2023-10-04T01:31:15.524 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383075 -le 1696383282 ]] 2023-10-04T01:31:15.524 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:15.525 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:15.525 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:25.526 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:25.527 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:25.528 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:25.535 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:26.381 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:26.382 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:26.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383086 2023-10-04T01:31:26.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383086 -le 1696383282 ]] 2023-10-04T01:31:26.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:26.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:26.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:36.391 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:36.395 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:36.398 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:36.417 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:37.260 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:37.260 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:37.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383097 2023-10-04T01:31:37.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383097 -le 1696383282 ]] 2023-10-04T01:31:37.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:37.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:37.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:47.272 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:47.272 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:47.291 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:47.291 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:48.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:48.161 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:48.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383108 2023-10-04T01:31:48.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383108 -le 1696383282 ]] 2023-10-04T01:31:48.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:48.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:48.162 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:31:52.325 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:52.320+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:31:52.325 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:52.320+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:31:52.325 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:52.320+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:31:56.950 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:56.944+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:31:56.950 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:31:56.944+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:31:58.168 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:31:58.172 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:31:58.175 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:31:58.194 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:31:59.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:31:59.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:31:59.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383119 2023-10-04T01:31:59.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383119 -le 1696383282 ]] 2023-10-04T01:31:59.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:31:59.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:31:59.075 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:32:09.082 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:32:09.083 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:32:09.083 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:32:09.095 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:32:10.070 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:32:10.070 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:32:10.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383130 2023-10-04T01:32:10.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383130 -le 1696383282 ]] 2023-10-04T01:32:10.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:32:10.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:32:10.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:32:20.073 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:32:20.074 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:32:20.082 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:32:20.087 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:32:21.033 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:32:21.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:32:21.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383141 2023-10-04T01:32:21.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383141 -le 1696383282 ]] 2023-10-04T01:32:21.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:32:21.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:32:21.034 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:32:24.653 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.644+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:32:24.659 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.652+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:32:24.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.668+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:32:24.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.668+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:32:24.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.668+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:32:24.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.668+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:32:24.675 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:24.668+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:32:31.043 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:32:31.044 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:32:31.046 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:32:31.049 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:32:32.109 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:32:32.109 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:32:32.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383152 2023-10-04T01:32:32.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383152 -le 1696383282 ]] 2023-10-04T01:32:32.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:32:32.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:32:32.110 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:32:32.891 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.887+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:32:32.891 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.887+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:32:32.901 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:32:32.901 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:32:32.902 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:32:32.903 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:32:32.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:32:39.462 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:32:39.455+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:32:39.466 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:32:39.455+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:32:42.112 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:32:42.112 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:32:42.114 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:32:42.123 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:32:43.029 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:32:43.030 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:32:43.030 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383163 2023-10-04T01:32:43.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383163 -le 1696383282 ]] 2023-10-04T01:32:43.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:32:43.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:32:43.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:32:53.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:32:53.033 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:32:53.039 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:32:53.047 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:32:54.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:32:54.016 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:32:54.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383174 2023-10-04T01:32:54.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383174 -le 1696383282 ]] 2023-10-04T01:32:54.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:32:54.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:32:54.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:04.020 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:04.024 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:04.028 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:04.031 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:04.947 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:04.963 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:04.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383184 2023-10-04T01:33:04.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383184 -le 1696383282 ]] 2023-10-04T01:33:04.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:04.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:04.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:14.500 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:33:14.494+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:33:14.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:14.982 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:14.983 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:15.001 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:15.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:15.939 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:15.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383195 2023-10-04T01:33:15.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383195 -le 1696383282 ]] 2023-10-04T01:33:15.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:15.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:15.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:25.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:25.945 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:25.947 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:25.952 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:26.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:26.869 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:26.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383206 2023-10-04T01:33:26.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383206 -le 1696383282 ]] 2023-10-04T01:33:26.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:26.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:26.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:32.440 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:33:32.426+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:33:36.872 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:36.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:36.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:36.874 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:37.798 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:37.798 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:37.799 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383217 2023-10-04T01:33:37.799 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383217 -le 1696383282 ]] 2023-10-04T01:33:37.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:37.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:37.800 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:47.805 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:47.807 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:47.814 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:47.815 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:48.658 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:48.658 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:48.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383228 2023-10-04T01:33:48.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383228 -le 1696383282 ]] 2023-10-04T01:33:48.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:48.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:48.659 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:33:56.674 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:33:56.665+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:33:56.678 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:33:56.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:33:56.678 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:33:56.673+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:33:58.660 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:33:58.662 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:33:58.671 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:33:58.671 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:33:59.719 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:33:59.719 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:33:59.720 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383239 2023-10-04T01:33:59.720 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383239 -le 1696383282 ]] 2023-10-04T01:33:59.720 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:33:59.720 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:33:59.720 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:34:00.917 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:00.913+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:34:00.918 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:00.913+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:34:07.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:34:07.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:34:07.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:34:07.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:34:07.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:34:07.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:34:07.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:34:07.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:34:07.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:34:07.221 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:34:07.217+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:34:09.724 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:34:09.725 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:34:09.728 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:34:09.738 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:34:10.644 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:34:10.644 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:34:10.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383250 2023-10-04T01:34:10.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383250 -le 1696383282 ]] 2023-10-04T01:34:10.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:34:10.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:34:10.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:34:20.651 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:34:20.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:34:20.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:34:20.659 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:34:21.649 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:34:21.649 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:34:21.649 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383261 2023-10-04T01:34:21.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383261 -le 1696383282 ]] 2023-10-04T01:34:21.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:34:21.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:34:21.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:34:25.489 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:34:25.481+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:34:25.489 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:34:25.481+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:34:31.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:34:31.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:34:31.652 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:34:31.661 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:34:32.547 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:34:32.547 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:34:32.548 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383272 2023-10-04T01:34:32.548 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383272 -le 1696383282 ]] 2023-10-04T01:34:32.549 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ ! -z 172.21.15.27:0/3787056471 ]] 2023-10-04T01:34:32.549 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 172.21.15.27:0/3787056471 != \1\7\2\.\2\1\.\1\5\.\2\7\:\0\/\3\7\8\7\0\5\6\4\7\1 ]] 2023-10-04T01:34:32.549 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:34:42.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq '.active_clients[]' 2023-10-04T01:34:42.551 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-04T01:34:42.554 INFO:tasks.workunit.client.0.smithi027.stderr:++ ceph mgr dump 2023-10-04T01:34:42.560 INFO:tasks.workunit.client.0.smithi027.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-04T01:34:43.417 INFO:tasks.workunit.client.0.smithi027.stderr:+ CLIENT_ADDR=172.21.15.27:0/3787056471 2023-10-04T01:34:43.419 INFO:tasks.workunit.client.0.smithi027.stderr:++ date +%s 2023-10-04T01:34:43.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ CURRENT_TIME=1696383283 2023-10-04T01:34:43.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ [[ 1696383283 -le 1696383282 ]] 2023-10-04T01:34:43.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i = 1 )) 2023-10-04T01:34:43.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:34:43.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:34:43.741 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:34:43.736+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:34:43.742 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:34:43.814 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:34:53.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:34:53.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:34:53.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:34:53.889 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:34:53.884+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:34:53.890 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:34:53.902 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:03.914 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:03.914 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:03.914 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:03.995 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:03.992+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:03.995 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:04.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:13.704 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:13.692+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:35:13.708 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:13.700+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:35:13.708 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:13.700+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:35:14.012 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:14.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:14.013 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:14.118 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:14.112+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:14.118 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:14.129 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:24.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:24.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:24.131 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:24.225 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:24.219+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:24.226 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:24.233 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:26.873 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.867+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:35:26.894 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.867+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:35:26.894 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.875+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:35:26.904 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:35:26.905 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:35:26.895+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:35:34.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:34.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:34.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:34.337 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:34.331+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:34.337 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:34.351 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:44.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:44.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:44.353 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:44.484 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:44.479+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:44.485 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:44.499 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:35:54.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:35:54.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:35:54.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:35:54.613 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:35:54.607+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:35:54.613 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:35:54.631 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:04.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:04.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:04.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:04.764 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:04.758+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:04.765 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:04.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:14.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:14.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:14.783 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:14.885 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:14.878+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:14.886 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:14.897 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:24.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:24.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:24.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:25.006 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:24.998+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:25.006 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:25.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:35.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:35.139 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:35.139 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:35.133+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:35.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:45.173 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:45.173 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:45.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:45.253 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:45.249+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:45.254 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:45.270 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:36:55.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:36:55.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:36:55.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:36:55.356 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:36:55.353+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:36:55.356 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:36:55.381 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:05.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:05.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:05.384 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:05.443 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:05.441+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:05.444 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:05.448 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:15.463 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:15.463 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:15.463 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:15.522 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:15.516+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:15.527 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:15.554 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:25.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:25.555 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:25.556 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:25.647 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:25.640+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:25.647 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:25.656 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:35.658 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:35.658 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:35.658 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:35.723 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:35.716+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:35.723 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:35.730 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:45.731 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:45.732 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:45.732 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:45.820 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:45.816+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:45.821 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:45.828 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:37:48.360 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.356+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:37:48.360 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.356+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:37:48.360 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.356+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:37:48.366 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.360+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:37:48.366 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.360+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:37:48.372 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.368+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:37:48.372 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:48.368+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:37:50.080 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.072+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:37:50.081 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.072+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:37:50.083 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.076+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:37:50.100 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.096+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:37:50.100 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.096+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:37:50.100 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.096+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:37:50.100 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:37:50.096+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:37:55.830 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:37:55.830 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:37:55.830 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:37:55.930 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:37:55.923+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:37:55.930 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:37:55.946 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:38:04.607 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:38:04.599+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:38:04.608 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:38:04.599+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:38:04.608 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:38:04.599+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:38:05.949 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:38:05.949 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:38:05.949 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:38:06.053 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:38:06.047+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:38:06.055 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:38:06.064 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:38:16.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:38:16.066 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:38:16.066 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:38:16.140 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:38:16.142 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:38:16.135+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:38:16.154 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:38:23.445 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:23.435+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:38:23.445 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:23.435+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:38:26.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:38:26.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:38:26.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:38:26.264 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:38:26.259+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:38:26.265 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:38:26.271 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:38:31.145 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:31.138+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:38:31.154 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:31.146+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:38:31.155 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:31.146+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:38:31.155 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:38:31.146+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:38:36.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:38:36.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:38:36.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-04T01:38:36.373 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:38:36.366+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:38:36.374 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:38:36.390 INFO:tasks.workunit.client.0.smithi027.stderr:+ sleep 10 2023-10-04T01:38:46.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i++ )) 2023-10-04T01:38:46.405 INFO:tasks.workunit.client.0.smithi027.stderr:+ (( i <= 24 )) 2023-10-04T01:38:46.405 INFO:tasks.workunit.client.0.smithi027.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-04T01:38:46.405 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'every 2m' 2023-10-04T01:38:46.502 INFO:tasks.ceph.mgr.x.smithi027.stderr:2023-10-04T01:38:46.494+0000 7ff17b16e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-04T01:38:46.503 INFO:tasks.workunit.client.0.smithi027.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-04T01:38:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T01:38:46.535 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-04T01:38:46.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-04T01:38:47.257 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-04T01:38:47.342 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=85a3acddc0494e7881b7e5ac249b3474 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ajarr_ceph_7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-04T01:38:47.345 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-04T01:38:47.394 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-04T01:38:47.403 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-04T01:38:47.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:38:48.014 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:38:48.015 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:38:48.044 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2106,"stamp":"2023-10-04T01:38:47.571875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26865156665,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1937263,"num_read_kb":6962385,"num_write":1777065,"num_write_kb":6941799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95045,"ondisk_log_size":95045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18997336,"kb_used_data":13782208,"kb_used_omap":4113,"kb_used_meta":5210990,"kb_avail":264118184,"statfs":{"total":289910292480,"available":270457020416,"internally_reserved":0,"allocated":14112980992,"data_stored":33795776884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5336053964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,4,15,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17932288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4271,"num_read_kb":17048,"num_write":4348,"num_write_kb":17392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"2.7","version":"140'7465","reported_seq":10594,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.365747+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T01:38:40.365747+0000","last_peered":"2023-10-04T01:38:40.365747+0000","last_clean":"2023-10-04T01:38:40.365747+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:38:40.365747+0000","last_undegraded":"2023-10-04T01:38:40.365747+0000","last_fullsized":"2023-10-04T01:38:40.365747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:24:19.202814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'6976","reported_seq":10531,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.138857+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T01:38:40.138857+0000","last_peered":"2023-10-04T01:38:40.138857+0000","last_clean":"2023-10-04T01:38:40.138857+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:38:40.138857+0000","last_undegraded":"2023-10-04T01:38:40.138857+0000","last_fullsized":"2023-10-04T01:38:40.138857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6976,"log_dups_size":0,"ondisk_log_size":6976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:36:25.294615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'7600","reported_seq":11036,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363488+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T01:38:40.363488+0000","last_peered":"2023-10-04T01:38:40.363488+0000","last_clean":"2023-10-04T01:38:40.363488+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:38:40.363488+0000","last_undegraded":"2023-10-04T01:38:40.363488+0000","last_fullsized":"2023-10-04T01:38:40.363488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:56:40.215197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6027","reported_seq":10309,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363554+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T01:38:40.363554+0000","last_peered":"2023-10-04T01:38:40.363554+0000","last_clean":"2023-10-04T01:38:40.363554+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:38:40.363554+0000","last_undegraded":"2023-10-04T01:38:40.363554+0000","last_fullsized":"2023-10-04T01:38:40.363554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:58:56.013414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'439291","reported_seq":867258,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.844304+0000","last_change":"2023-10-04T01:07:45.949637+0000","last_active":"2023-10-04T01:38:44.844304+0000","last_peered":"2023-10-04T01:38:44.844304+0000","last_clean":"2023-10-04T01:38:44.844304+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:38:44.844304+0000","last_undegraded":"2023-10-04T01:38:44.844304+0000","last_fullsized":"2023-10-04T01:38:44.844304+0000","mapping_epoch":15,"log_start":"139'429271","ondisk_log_start":"139'429271","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:56:39.843606+0000","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6745788416,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":433386,"num_read_kb":1703512,"num_write":432664,"num_write_kb":1730656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'15864","reported_seq":20668,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:42.097969+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T01:38:42.097969+0000","last_peered":"2023-10-04T01:38:42.097969+0000","last_clean":"2023-10-04T01:38:42.097969+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:38:42.097969+0000","last_undegraded":"2023-10-04T01:38:42.097969+0000","last_fullsized":"2023-10-04T01:38:42.097969+0000","mapping_epoch":12,"log_start":"77'5800","ondisk_log_start":"77'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:23:11.908717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'429768","reported_seq":848598,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.557291+0000","last_change":"2023-10-04T01:07:37.244061+0000","last_active":"2023-10-04T01:38:43.557291+0000","last_peered":"2023-10-04T01:38:43.557291+0000","last_clean":"2023-10-04T01:38:43.557291+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:38:43.557291+0000","last_undegraded":"2023-10-04T01:38:43.557291+0000","last_fullsized":"2023-10-04T01:38:43.557291+0000","mapping_epoch":15,"log_start":"139'419760","ondisk_log_start":"139'419760","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:01:36.035976+0000","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6605557760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":424149,"num_read_kb":1667312,"num_write":423267,"num_write_kb":1693068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"136'2062","reported_seq":8529,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.263239+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T01:38:45.263239+0000","last_peered":"2023-10-04T01:38:45.263239+0000","last_clean":"2023-10-04T01:38:45.263239+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:38:45.263239+0000","last_undegraded":"2023-10-04T01:38:45.263239+0000","last_fullsized":"2023-10-04T01:38:45.263239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'449016","reported_seq":887585,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.843221+0000","last_change":"2023-10-04T01:07:40.886698+0000","last_active":"2023-10-04T01:38:44.843221+0000","last_peered":"2023-10-04T01:38:44.843221+0000","last_clean":"2023-10-04T01:38:44.843221+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:38:44.843221+0000","last_undegraded":"2023-10-04T01:38:44.843221+0000","last_fullsized":"2023-10-04T01:38:44.843221+0000","mapping_epoch":15,"log_start":"139'438964","ondisk_log_start":"139'438964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:54:45.511629+0000","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6911299584,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":444126,"num_read_kb":1746552,"num_write":442250,"num_write_kb":1769000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"136'4501","reported_seq":9188,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.265847+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T01:38:45.265847+0000","last_peered":"2023-10-04T01:38:45.265847+0000","last_clean":"2023-10-04T01:38:45.265847+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:38:45.265847+0000","last_undegraded":"2023-10-04T01:38:45.265847+0000","last_fullsized":"2023-10-04T01:38:45.265847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'429752","reported_seq":848849,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.587751+0000","last_change":"2023-10-04T01:07:43.400960+0000","last_active":"2023-10-04T01:38:43.587751+0000","last_peered":"2023-10-04T01:38:43.587751+0000","last_clean":"2023-10-04T01:38:43.587751+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:38:43.587751+0000","last_undegraded":"2023-10-04T01:38:43.587751+0000","last_fullsized":"2023-10-04T01:38:43.587751+0000","mapping_epoch":15,"log_start":"139'419663","ondisk_log_start":"139'419663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:09:14.439644+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6601904147,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":424428,"num_read_kb":1668156,"num_write":423240,"num_write_kb":1692954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'16784","reported_seq":21813,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.430332+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T01:38:40.430332+0000","last_peered":"2023-10-04T01:38:40.430332+0000","last_clean":"2023-10-04T01:38:40.430332+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:38:40.430332+0000","last_undegraded":"2023-10-04T01:38:40.430332+0000","last_fullsized":"2023-10-04T01:38:40.430332+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:17:16.511145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":299,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-04T01:37:38.228044+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T01:37:38.228044+0000","last_peered":"2023-10-04T01:37:38.228044+0000","last_clean":"2023-10-04T01:37:38.228044+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:37:38.228044+0000","last_undegraded":"2023-10-04T01:37:38.228044+0000","last_fullsized":"2023-10-04T01:37:38.228044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26864549907,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1726089,"num_read_kb":6785532,"num_write":1721421,"num_write_kb":6885678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14107688960,"data_stored":33794785318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":54779,"ondisk_log_size":54779,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8502828,"kb_used_data":6892132,"kb_used_omap":2097,"kb_used_meta":1608590,"kb_avail":85869012,"statfs":{"total":96636764160,"available":87929868288,"internally_reserved":0,"allocated":7057543168,"data_stored":16899875071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1647196677},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35899999999999999,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":4.2510000000000003},"last":0.316},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42299999999999999,"15min":0.45200000000000001},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88100000000000001,"5min":2.145,"15min":4.2869999999999999},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:38:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":2.492,"15min":6.7709999999999999},"last":0.371},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.442,"15min":0.501},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":7.415,"15min":7.415},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5212952,"kb_used_data":3388752,"kb_used_omap":997,"kb_used_meta":1823194,"kb_avail":89158888,"statfs":{"total":96636764160,"available":91298701312,"internally_reserved":0,"allocated":3470082048,"data_stored":8319911567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1866951067},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39800000000000002,"15min":0.376},"min":{"1min":0.187,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2810000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.55300000000000005,"15min":0.50800000000000001},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3079999999999998,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.68400000000000005},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.18099999999999999},"max":{"1min":3.919,"5min":14.467000000000001,"15min":14.467000000000001},"last":0.44},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.68200000000000005,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71799999999999997,"5min":14.503,"15min":14.503},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739156,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5281556,"kb_used_data":3501324,"kb_used_omap":1018,"kb_used_meta":1779205,"kb_avail":89090284,"statfs":{"total":96636764160,"available":91228450816,"internally_reserved":0,"allocated":3585355776,"data_stored":8575990246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1821906220},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.6400000000000001,"15min":1.2090000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.032,"5min":207.047,"15min":207.047},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.37,"5min":2.6899999999999999,"15min":1.2370000000000001},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.36299999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:38:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.191},"max":{"1min":5.4710000000000001,"5min":5.907,"15min":7.3300000000000001},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3583651840,"data_stored":8575627264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3467292672,"data_stored":8318865427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7056744448,"data_stored":16900292627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:38:48.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:38:48.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:38:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:38:48.545 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2106,"stamp":"2023-10-04T01:38:47.571875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26865156665,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1937263,"num_read_kb":6962385,"num_write":1777065,"num_write_kb":6941799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95045,"ondisk_log_size":95045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18997336,"kb_used_data":13782208,"kb_used_omap":4113,"kb_used_meta":5210990,"kb_avail":264118184,"statfs":{"total":289910292480,"available":270457020416,"internally_reserved":0,"allocated":14112980992,"data_stored":33795776884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5336053964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,4,15,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17932288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4271,"num_read_kb":17048,"num_write":4348,"num_write_kb":17392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"2.7","version":"140'7465","reported_seq":10594,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.365747+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T01:38:40.365747+0000","last_peered":"2023-10-04T01:38:40.365747+0000","last_clean":"2023-10-04T01:38:40.365747+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:38:40.365747+0000","last_undegraded":"2023-10-04T01:38:40.365747+0000","last_fullsized":"2023-10-04T01:38:40.365747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:24:19.202814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'6976","reported_seq":10531,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.138857+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T01:38:40.138857+0000","last_peered":"2023-10-04T01:38:40.138857+0000","last_clean":"2023-10-04T01:38:40.138857+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:38:40.138857+0000","last_undegraded":"2023-10-04T01:38:40.138857+0000","last_fullsized":"2023-10-04T01:38:40.138857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6976,"log_dups_size":0,"ondisk_log_size":6976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:36:25.294615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'7600","reported_seq":11036,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363488+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T01:38:40.363488+0000","last_peered":"2023-10-04T01:38:40.363488+0000","last_clean":"2023-10-04T01:38:40.363488+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:38:40.363488+0000","last_undegraded":"2023-10-04T01:38:40.363488+0000","last_fullsized":"2023-10-04T01:38:40.363488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:56:40.215197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6027","reported_seq":10309,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363554+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T01:38:40.363554+0000","last_peered":"2023-10-04T01:38:40.363554+0000","last_clean":"2023-10-04T01:38:40.363554+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:38:40.363554+0000","last_undegraded":"2023-10-04T01:38:40.363554+0000","last_fullsized":"2023-10-04T01:38:40.363554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:58:56.013414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'439291","reported_seq":867258,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.844304+0000","last_change":"2023-10-04T01:07:45.949637+0000","last_active":"2023-10-04T01:38:44.844304+0000","last_peered":"2023-10-04T01:38:44.844304+0000","last_clean":"2023-10-04T01:38:44.844304+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:38:44.844304+0000","last_undegraded":"2023-10-04T01:38:44.844304+0000","last_fullsized":"2023-10-04T01:38:44.844304+0000","mapping_epoch":15,"log_start":"139'429271","ondisk_log_start":"139'429271","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:56:39.843606+0000","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6745788416,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":433386,"num_read_kb":1703512,"num_write":432664,"num_write_kb":1730656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'15864","reported_seq":20668,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:42.097969+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T01:38:42.097969+0000","last_peered":"2023-10-04T01:38:42.097969+0000","last_clean":"2023-10-04T01:38:42.097969+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:38:42.097969+0000","last_undegraded":"2023-10-04T01:38:42.097969+0000","last_fullsized":"2023-10-04T01:38:42.097969+0000","mapping_epoch":12,"log_start":"77'5800","ondisk_log_start":"77'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:23:11.908717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'429768","reported_seq":848598,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.557291+0000","last_change":"2023-10-04T01:07:37.244061+0000","last_active":"2023-10-04T01:38:43.557291+0000","last_peered":"2023-10-04T01:38:43.557291+0000","last_clean":"2023-10-04T01:38:43.557291+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:38:43.557291+0000","last_undegraded":"2023-10-04T01:38:43.557291+0000","last_fullsized":"2023-10-04T01:38:43.557291+0000","mapping_epoch":15,"log_start":"139'419760","ondisk_log_start":"139'419760","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:01:36.035976+0000","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6605557760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":424149,"num_read_kb":1667312,"num_write":423267,"num_write_kb":1693068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"136'2062","reported_seq":8529,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.263239+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T01:38:45.263239+0000","last_peered":"2023-10-04T01:38:45.263239+0000","last_clean":"2023-10-04T01:38:45.263239+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:38:45.263239+0000","last_undegraded":"2023-10-04T01:38:45.263239+0000","last_fullsized":"2023-10-04T01:38:45.263239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'449016","reported_seq":887585,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.843221+0000","last_change":"2023-10-04T01:07:40.886698+0000","last_active":"2023-10-04T01:38:44.843221+0000","last_peered":"2023-10-04T01:38:44.843221+0000","last_clean":"2023-10-04T01:38:44.843221+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:38:44.843221+0000","last_undegraded":"2023-10-04T01:38:44.843221+0000","last_fullsized":"2023-10-04T01:38:44.843221+0000","mapping_epoch":15,"log_start":"139'438964","ondisk_log_start":"139'438964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:54:45.511629+0000","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6911299584,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":444126,"num_read_kb":1746552,"num_write":442250,"num_write_kb":1769000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"136'4501","reported_seq":9188,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.265847+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T01:38:45.265847+0000","last_peered":"2023-10-04T01:38:45.265847+0000","last_clean":"2023-10-04T01:38:45.265847+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:38:45.265847+0000","last_undegraded":"2023-10-04T01:38:45.265847+0000","last_fullsized":"2023-10-04T01:38:45.265847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'429752","reported_seq":848849,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.587751+0000","last_change":"2023-10-04T01:07:43.400960+0000","last_active":"2023-10-04T01:38:43.587751+0000","last_peered":"2023-10-04T01:38:43.587751+0000","last_clean":"2023-10-04T01:38:43.587751+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:38:43.587751+0000","last_undegraded":"2023-10-04T01:38:43.587751+0000","last_fullsized":"2023-10-04T01:38:43.587751+0000","mapping_epoch":15,"log_start":"139'419663","ondisk_log_start":"139'419663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:09:14.439644+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6601904147,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":424428,"num_read_kb":1668156,"num_write":423240,"num_write_kb":1692954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'16784","reported_seq":21813,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.430332+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T01:38:40.430332+0000","last_peered":"2023-10-04T01:38:40.430332+0000","last_clean":"2023-10-04T01:38:40.430332+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:38:40.430332+0000","last_undegraded":"2023-10-04T01:38:40.430332+0000","last_fullsized":"2023-10-04T01:38:40.430332+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:17:16.511145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":299,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-04T01:37:38.228044+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T01:37:38.228044+0000","last_peered":"2023-10-04T01:37:38.228044+0000","last_clean":"2023-10-04T01:37:38.228044+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:37:38.228044+0000","last_undegraded":"2023-10-04T01:37:38.228044+0000","last_fullsized":"2023-10-04T01:37:38.228044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26864549907,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1726089,"num_read_kb":6785532,"num_write":1721421,"num_write_kb":6885678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14107688960,"data_stored":33794785318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":54779,"ondisk_log_size":54779,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8502828,"kb_used_data":6892132,"kb_used_omap":2097,"kb_used_meta":1608590,"kb_avail":85869012,"statfs":{"total":96636764160,"available":87929868288,"internally_reserved":0,"allocated":7057543168,"data_stored":16899875071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1647196677},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35899999999999999,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":4.2510000000000003},"last":0.316},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42299999999999999,"15min":0.45200000000000001},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88100000000000001,"5min":2.145,"15min":4.2869999999999999},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:38:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":2.492,"15min":6.7709999999999999},"last":0.371},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.442,"15min":0.501},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":7.415,"15min":7.415},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5212952,"kb_used_data":3388752,"kb_used_omap":997,"kb_used_meta":1823194,"kb_avail":89158888,"statfs":{"total":96636764160,"available":91298701312,"internally_reserved":0,"allocated":3470082048,"data_stored":8319911567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1866951067},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39800000000000002,"15min":0.376},"min":{"1min":0.187,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2810000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.55300000000000005,"15min":0.50800000000000001},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3079999999999998,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.68400000000000005},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.18099999999999999},"max":{"1min":3.919,"5min":14.467000000000001,"15min":14.467000000000001},"last":0.44},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.68200000000000005,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71799999999999997,"5min":14.503,"15min":14.503},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739156,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5281556,"kb_used_data":3501324,"kb_used_omap":1018,"kb_used_meta":1779205,"kb_avail":89090284,"statfs":{"total":96636764160,"available":91228450816,"internally_reserved":0,"allocated":3585355776,"data_stored":8575990246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1821906220},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.6400000000000001,"15min":1.2090000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.032,"5min":207.047,"15min":207.047},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.37,"5min":2.6899999999999999,"15min":1.2370000000000001},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.36299999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:38:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.191},"max":{"1min":5.4710000000000001,"5min":5.907,"15min":7.3300000000000001},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3583651840,"data_stored":8575627264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3467292672,"data_stored":8318865427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7056744448,"data_stored":16900292627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:38:48.546 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-04T01:38:48.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:38:49.050 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:38:49.051 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:38:49.075 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2106,"stamp":"2023-10-04T01:38:47.571875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26865156665,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1937263,"num_read_kb":6962385,"num_write":1777065,"num_write_kb":6941799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95045,"ondisk_log_size":95045,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18997336,"kb_used_data":13782208,"kb_used_omap":4113,"kb_used_meta":5210990,"kb_avail":264118184,"statfs":{"total":289910292480,"available":270457020416,"internally_reserved":0,"allocated":14112980992,"data_stored":33795776884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5336053964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,4,15,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17932288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4271,"num_read_kb":17048,"num_write":4348,"num_write_kb":17392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"2.7","version":"140'7465","reported_seq":10594,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.365747+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T01:38:40.365747+0000","last_peered":"2023-10-04T01:38:40.365747+0000","last_clean":"2023-10-04T01:38:40.365747+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:38:40.365747+0000","last_undegraded":"2023-10-04T01:38:40.365747+0000","last_fullsized":"2023-10-04T01:38:40.365747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:24:19.202814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'6976","reported_seq":10531,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.138857+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T01:38:40.138857+0000","last_peered":"2023-10-04T01:38:40.138857+0000","last_clean":"2023-10-04T01:38:40.138857+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:38:40.138857+0000","last_undegraded":"2023-10-04T01:38:40.138857+0000","last_fullsized":"2023-10-04T01:38:40.138857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6976,"log_dups_size":0,"ondisk_log_size":6976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:36:25.294615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'7600","reported_seq":11036,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363488+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T01:38:40.363488+0000","last_peered":"2023-10-04T01:38:40.363488+0000","last_clean":"2023-10-04T01:38:40.363488+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:38:40.363488+0000","last_undegraded":"2023-10-04T01:38:40.363488+0000","last_fullsized":"2023-10-04T01:38:40.363488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:56:40.215197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6027","reported_seq":10309,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.363554+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T01:38:40.363554+0000","last_peered":"2023-10-04T01:38:40.363554+0000","last_clean":"2023-10-04T01:38:40.363554+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:38:40.363554+0000","last_undegraded":"2023-10-04T01:38:40.363554+0000","last_fullsized":"2023-10-04T01:38:40.363554+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:58:56.013414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'439291","reported_seq":867258,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.844304+0000","last_change":"2023-10-04T01:07:45.949637+0000","last_active":"2023-10-04T01:38:44.844304+0000","last_peered":"2023-10-04T01:38:44.844304+0000","last_clean":"2023-10-04T01:38:44.844304+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:38:44.844304+0000","last_undegraded":"2023-10-04T01:38:44.844304+0000","last_fullsized":"2023-10-04T01:38:44.844304+0000","mapping_epoch":15,"log_start":"139'429271","ondisk_log_start":"139'429271","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:56:39.843606+0000","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6745788416,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":433386,"num_read_kb":1703512,"num_write":432664,"num_write_kb":1730656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'15864","reported_seq":20668,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:42.097969+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T01:38:42.097969+0000","last_peered":"2023-10-04T01:38:42.097969+0000","last_clean":"2023-10-04T01:38:42.097969+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:38:42.097969+0000","last_undegraded":"2023-10-04T01:38:42.097969+0000","last_fullsized":"2023-10-04T01:38:42.097969+0000","mapping_epoch":12,"log_start":"77'5800","ondisk_log_start":"77'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:23:11.908717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'429768","reported_seq":848598,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.557291+0000","last_change":"2023-10-04T01:07:37.244061+0000","last_active":"2023-10-04T01:38:43.557291+0000","last_peered":"2023-10-04T01:38:43.557291+0000","last_clean":"2023-10-04T01:38:43.557291+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:38:43.557291+0000","last_undegraded":"2023-10-04T01:38:43.557291+0000","last_fullsized":"2023-10-04T01:38:43.557291+0000","mapping_epoch":15,"log_start":"139'419760","ondisk_log_start":"139'419760","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:01:36.035976+0000","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6605557760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":424149,"num_read_kb":1667312,"num_write":423267,"num_write_kb":1693068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"136'2062","reported_seq":8529,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.263239+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T01:38:45.263239+0000","last_peered":"2023-10-04T01:38:45.263239+0000","last_clean":"2023-10-04T01:38:45.263239+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:38:45.263239+0000","last_undegraded":"2023-10-04T01:38:45.263239+0000","last_fullsized":"2023-10-04T01:38:45.263239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'449016","reported_seq":887585,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:44.843221+0000","last_change":"2023-10-04T01:07:40.886698+0000","last_active":"2023-10-04T01:38:44.843221+0000","last_peered":"2023-10-04T01:38:44.843221+0000","last_clean":"2023-10-04T01:38:44.843221+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:38:44.843221+0000","last_undegraded":"2023-10-04T01:38:44.843221+0000","last_fullsized":"2023-10-04T01:38:44.843221+0000","mapping_epoch":15,"log_start":"139'438964","ondisk_log_start":"139'438964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:54:45.511629+0000","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6911299584,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":444126,"num_read_kb":1746552,"num_write":442250,"num_write_kb":1769000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"136'4501","reported_seq":9188,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.265847+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T01:38:45.265847+0000","last_peered":"2023-10-04T01:38:45.265847+0000","last_clean":"2023-10-04T01:38:45.265847+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:38:45.265847+0000","last_undegraded":"2023-10-04T01:38:45.265847+0000","last_fullsized":"2023-10-04T01:38:45.265847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'429752","reported_seq":848849,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:43.587751+0000","last_change":"2023-10-04T01:07:43.400960+0000","last_active":"2023-10-04T01:38:43.587751+0000","last_peered":"2023-10-04T01:38:43.587751+0000","last_clean":"2023-10-04T01:38:43.587751+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:38:43.587751+0000","last_undegraded":"2023-10-04T01:38:43.587751+0000","last_fullsized":"2023-10-04T01:38:43.587751+0000","mapping_epoch":15,"log_start":"139'419663","ondisk_log_start":"139'419663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:09:14.439644+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6601904147,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":424428,"num_read_kb":1668156,"num_write":423240,"num_write_kb":1692954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'16784","reported_seq":21813,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:40.430332+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T01:38:40.430332+0000","last_peered":"2023-10-04T01:38:40.430332+0000","last_clean":"2023-10-04T01:38:40.430332+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:38:40.430332+0000","last_undegraded":"2023-10-04T01:38:40.430332+0000","last_fullsized":"2023-10-04T01:38:40.430332+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:17:16.511145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":299,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-04T01:37:38.228044+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T01:37:38.228044+0000","last_peered":"2023-10-04T01:37:38.228044+0000","last_clean":"2023-10-04T01:37:38.228044+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:37:38.228044+0000","last_undegraded":"2023-10-04T01:37:38.228044+0000","last_fullsized":"2023-10-04T01:37:38.228044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26864549907,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1726089,"num_read_kb":6785532,"num_write":1721421,"num_write_kb":6885678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14107688960,"data_stored":33794785318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":54779,"ondisk_log_size":54779,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8502828,"kb_used_data":6892132,"kb_used_omap":2097,"kb_used_meta":1608590,"kb_avail":85869012,"statfs":{"total":96636764160,"available":87929868288,"internally_reserved":0,"allocated":7057543168,"data_stored":16899875071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1647196677},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35899999999999999,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":4.2510000000000003},"last":0.316},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42299999999999999,"15min":0.45200000000000001},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88100000000000001,"5min":2.145,"15min":4.2869999999999999},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:38:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":2.492,"15min":6.7709999999999999},"last":0.371},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.442,"15min":0.501},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":7.415,"15min":7.415},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5212952,"kb_used_data":3388752,"kb_used_omap":997,"kb_used_meta":1823194,"kb_avail":89158888,"statfs":{"total":96636764160,"available":91298701312,"internally_reserved":0,"allocated":3470082048,"data_stored":8319911567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1866951067},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,9,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39800000000000002,"15min":0.376},"min":{"1min":0.187,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2810000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.55300000000000005,"15min":0.50800000000000001},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3079999999999998,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.68400000000000005},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.18099999999999999},"max":{"1min":3.919,"5min":14.467000000000001,"15min":14.467000000000001},"last":0.44},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.68200000000000005,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71799999999999997,"5min":14.503,"15min":14.503},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739156,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5281556,"kb_used_data":3501324,"kb_used_omap":1018,"kb_used_meta":1779205,"kb_avail":89090284,"statfs":{"total":96636764160,"available":91228450816,"internally_reserved":0,"allocated":3585355776,"data_stored":8575990246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1821906220},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.6400000000000001,"15min":1.2090000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.032,"5min":207.047,"15min":207.047},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.37,"5min":2.6899999999999999,"15min":1.2370000000000001},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.36299999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:38:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.191},"max":{"1min":5.4710000000000001,"5min":5.907,"15min":7.3300000000000001},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3583651840,"data_stored":8575627264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3467292672,"data_stored":8318865427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7056744448,"data_stored":16900292627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:38:49.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-04T01:38:49.576 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:38:49.576 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":140,"fsid":"250ac99b-0ea2-47d3-80de-42417fc97a77","created":"2023-10-04T00:33:01.829402+0000","modified":"2023-10-04T01:37:48.040150+0000","last_up_change":"2023-10-04T00:33:11.628861+0000","last_in_change":"2023-10-04T00:33:02.817302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-04T00:33:12.795458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-04T00:33:17.315024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-04T00:33:23.002785+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":134,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a54a86-80fd-4af9-ad9b-9544a78b082c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6811","nonce":2975557568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6815","nonce":2975557568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6821","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6823","nonce":2975557568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6817","nonce":2975557568},{"type":"v1","addr":"172.21.15.27:6819","nonce":2975557568}]},"public_addr":"172.21.15.27:6811/2975557568","cluster_addr":"172.21.15.27:6815/2975557568","heartbeat_back_addr":"172.21.15.27:6823/2975557568","heartbeat_front_addr":"172.21.15.27:6819/2975557568","state":["exists","up"]},{"osd":1,"uuid":"d4cb688e-0dce-4f66-a7f8-ab84f8910e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6810","nonce":3555328205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6814","nonce":3555328205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6822","nonce":3555328205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":3555328205},{"type":"v1","addr":"172.21.15.27:6818","nonce":3555328205}]},"public_addr":"172.21.15.27:6810/3555328205","cluster_addr":"172.21.15.27:6814/3555328205","heartbeat_back_addr":"172.21.15.27:6822/3555328205","heartbeat_front_addr":"172.21.15.27:6818/3555328205","state":["exists","up"]},{"osd":2,"uuid":"b8b35e1a-6dd4-4e3e-81cd-4b5c06b523d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6801","nonce":3685667109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6803","nonce":3685667109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6807","nonce":3685667109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":3685667109},{"type":"v1","addr":"172.21.15.27:6805","nonce":3685667109}]},"public_addr":"172.21.15.27:6801/3685667109","cluster_addr":"172.21.15.27:6803/3685667109","heartbeat_back_addr":"172.21.15.27:6807/3685667109","heartbeat_front_addr":"172.21.15.27:6805/3685667109","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.399554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:07.881648+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-04T00:33:08.271297+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/3821044334":"2023-10-04T02:10:31.682648+0000","172.21.15.27:0/3677686417":"2023-10-04T02:09:22.668154+0000","172.21.15.27:0/2080516480":"2023-10-04T01:47:11.811289+0000","172.21.15.27:0/361536328":"2023-10-04T01:53:07.500627+0000","172.21.15.27:0/3787056471":"2023-10-04T02:11:40.491043+0000","172.21.15.27:0/3447838188":"2023-10-04T02:07:04.782913+0000","172.21.15.27:0/2013951676":"2023-10-04T01:41:15.163221+0000","172.21.15.27:0/2079221933":"2023-10-04T01:44:53.205681+0000","172.21.15.27:0/1535402043":"2023-10-04T01:57:53.686003+0000","172.21.15.27:0/4237805760":"2023-10-04T01:40:06.165600+0000","172.21.15.27:0/3301872187":"2023-10-04T01:48:20.391990+0000","172.21.15.27:0/1057421862":"2023-10-04T01:51:58.635745+0000","172.21.15.27:0/1678579581":"2023-10-04T01:50:49.426198+0000","172.21.15.27:0/2108081611":"2023-10-04T01:38:57.203892+0000","172.21.15.27:0/1990656485":"2023-10-04T01:59:02.806974+0000","172.21.15.27:0/978409279":"2023-10-04T01:46:02.137505+0000","172.21.15.27:0/1712573181":"2023-10-04T02:08:14.054041+0000","172.21.15.27:0/3739722863":"2023-10-04T02:04:46.558694+0000","172.21.15.27:0/1607046929":"2023-10-04T01:42:24.013449+0000","172.21.15.27:0/1582006596":"2023-10-04T02:03:38.150667+0000","172.21.15.27:0/303862463":"2023-10-04T01:54:16.432294+0000","172.21.15.27:0/928103956":"2023-10-04T01:43:44.272612+0000","172.21.15.27:0/1264225453":"2023-10-04T01:55:25.444600+0000","172.21.15.27:0/4009918629":"2023-10-04T01:56:34.058385+0000","172.21.15.27:0/2670833392":"2023-10-04T02:00:12.107073+0000","172.21.15.27:0/2949576082":"2023-10-04T02:06:06.863370+0000","172.21.15.27:0/3859805389":"2023-10-04T01:49:29.261992+0000","172.21.15.27:0/1712616634":"2023-10-04T02:01:20.364318+0000","172.21.15.27:0/503959746":"2023-10-04T02:02:29.327183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-04T01:38:50.610 INFO:tasks.ceph:Scrubbing osd.0 2023-10-04T01:38:50.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-04T01:38:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-04T01:38:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-04T01:38:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-04T01:38:50.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-04T01:38:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2023-10-04T01:38:51.427 INFO:tasks.ceph:Scrubbing osd.1 2023-10-04T01:38:51.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-04T01:38:51.732 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-04T01:38:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-04T01:38:51.732 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-04T01:38:51.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-04T01:38:52.241 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2023-10-04T01:38:52.290 INFO:tasks.ceph:Scrubbing osd.2 2023-10-04T01:38:52.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-04T01:38:52.568 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-04T01:38:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-04T01:38:52.569 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-04T01:38:52.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-04T01:38:53.210 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2023-10-04T01:38:53.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:38:53.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:38:53.782 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:38:53.811 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2109,"stamp":"2023-10-04T01:38:53.572767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26873741881,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1939174,"num_read_kb":6970013,"num_write":1779148,"num_write_kb":6950131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95161,"ondisk_log_size":95161,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19044308,"kb_used_data":13798652,"kb_used_omap":4113,"kb_used_meta":5241518,"kb_avail":264071212,"statfs":{"total":289910292480,"available":270408921088,"internally_reserved":0,"allocated":14129819648,"data_stored":33812615540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5367314636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,12,15,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17727488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4028,"num_read_kb":16084,"num_write":4300,"num_write_kb":17200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001897"},"pg_stats":[{"pgid":"2.7","version":"140'7471","reported_seq":10602,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:46.286921+0000","last_change":"2023-10-04T00:33:18.862801+0000","last_active":"2023-10-04T01:38:46.286921+0000","last_peered":"2023-10-04T01:38:46.286921+0000","last_clean":"2023-10-04T01:38:46.286921+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:38:46.286921+0000","last_undegraded":"2023-10-04T01:38:46.286921+0000","last_fullsized":"2023-10-04T01:38:46.286921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:24:19.202814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'6979","reported_seq":10535,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:45.365355+0000","last_change":"2023-10-04T00:33:18.861572+0000","last_active":"2023-10-04T01:38:45.365355+0000","last_peered":"2023-10-04T01:38:45.365355+0000","last_clean":"2023-10-04T01:38:45.365355+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:38:45.365355+0000","last_undegraded":"2023-10-04T01:38:45.365355+0000","last_fullsized":"2023-10-04T01:38:45.365355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:36:25.294615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'7606","reported_seq":11044,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:46.282541+0000","last_change":"2023-10-04T00:33:18.862802+0000","last_active":"2023-10-04T01:38:46.282541+0000","last_peered":"2023-10-04T01:38:46.282541+0000","last_clean":"2023-10-04T01:38:46.282541+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:38:46.282541+0000","last_undegraded":"2023-10-04T01:38:46.282541+0000","last_fullsized":"2023-10-04T01:38:46.282541+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":7606,"log_dups_size":0,"ondisk_log_size":7606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:56:40.215197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'6033","reported_seq":10317,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:46.282655+0000","last_change":"2023-10-04T00:33:18.861557+0000","last_active":"2023-10-04T01:38:46.282655+0000","last_peered":"2023-10-04T01:38:46.282655+0000","last_clean":"2023-10-04T01:38:46.282655+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:38:46.282655+0000","last_undegraded":"2023-10-04T01:38:46.282655+0000","last_fullsized":"2023-10-04T01:38:46.282655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:58:56.013414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"140'439820","reported_seq":868240,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:49.853868+0000","last_change":"2023-10-04T01:07:45.949637+0000","last_active":"2023-10-04T01:38:49.853868+0000","last_peered":"2023-10-04T01:38:49.853868+0000","last_clean":"2023-10-04T01:38:49.853868+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:38:49.853868+0000","last_undegraded":"2023-10-04T01:38:49.853868+0000","last_fullsized":"2023-10-04T01:38:49.853868+0000","mapping_epoch":15,"log_start":"139'429771","ondisk_log_start":"139'429771","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:56:39.843606+0000","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6747938816,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":433834,"num_read_kb":1705300,"num_write":433193,"num_write_kb":1732772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"140'15867","reported_seq":20674,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:47.056651+0000","last_change":"2023-10-04T00:33:18.866615+0000","last_active":"2023-10-04T01:38:47.056651+0000","last_peered":"2023-10-04T01:38:47.056651+0000","last_clean":"2023-10-04T01:38:47.056651+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:38:47.056651+0000","last_undegraded":"2023-10-04T01:38:47.056651+0000","last_fullsized":"2023-10-04T01:38:47.056651+0000","mapping_epoch":12,"log_start":"77'5800","ondisk_log_start":"77'5800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:23:11.908717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"140'430291","reported_seq":849603,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:48.593511+0000","last_change":"2023-10-04T01:07:37.244061+0000","last_active":"2023-10-04T01:38:48.593511+0000","last_peered":"2023-10-04T01:38:48.593511+0000","last_clean":"2023-10-04T01:38:48.593511+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:38:48.593511+0000","last_undegraded":"2023-10-04T01:38:48.593511+0000","last_fullsized":"2023-10-04T01:38:48.593511+0000","mapping_epoch":15,"log_start":"139'420260","ondisk_log_start":"139'420260","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T09:01:36.035976+0000","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6607720448,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":424631,"num_read_kb":1669236,"num_write":423790,"num_write_kb":1695160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"136'2062","reported_seq":8533,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:50.383240+0000","last_change":"2023-10-04T00:33:18.867015+0000","last_active":"2023-10-04T01:38:50.383240+0000","last_peered":"2023-10-04T01:38:50.383240+0000","last_clean":"2023-10-04T01:38:50.383240+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:38:50.383240+0000","last_undegraded":"2023-10-04T01:38:50.383240+0000","last_fullsized":"2023-10-04T01:38:50.383240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T11:09:44.017646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"140'449548","reported_seq":888611,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:49.865955+0000","last_change":"2023-10-04T01:07:40.886698+0000","last_active":"2023-10-04T01:38:49.865955+0000","last_peered":"2023-10-04T01:38:49.865955+0000","last_clean":"2023-10-04T01:38:49.865955+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:38:49.865955+0000","last_undegraded":"2023-10-04T01:38:49.865955+0000","last_fullsized":"2023-10-04T01:38:49.865955+0000","mapping_epoch":15,"log_start":"139'439464","ondisk_log_start":"139'439464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:54:45.511629+0000","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6913523712,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":444619,"num_read_kb":1748520,"num_write":442782,"num_write_kb":1771128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"136'4501","reported_seq":9191,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:50.383817+0000","last_change":"2023-10-04T00:33:18.866746+0000","last_active":"2023-10-04T01:38:50.383817+0000","last_peered":"2023-10-04T01:38:50.383817+0000","last_clean":"2023-10-04T01:38:50.383817+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:38:50.383817+0000","last_undegraded":"2023-10-04T01:38:50.383817+0000","last_fullsized":"2023-10-04T01:38:50.383817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:48:43.950154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"140'430251","reported_seq":849842,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:48.588699+0000","last_change":"2023-10-04T01:07:43.400960+0000","last_active":"2023-10-04T01:38:48.588699+0000","last_peered":"2023-10-04T01:38:48.588699+0000","last_clean":"2023-10-04T01:38:48.588699+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:38:48.588699+0000","last_undegraded":"2023-10-04T01:38:48.588699+0000","last_fullsized":"2023-10-04T01:38:48.588699+0000","mapping_epoch":15,"log_start":"139'420163","ondisk_log_start":"139'420163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:09:14.439644+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6603952147,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":424916,"num_read_kb":1670104,"num_write":423739,"num_write_kb":1694950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'16793","reported_seq":21825,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:46.315037+0000","last_change":"2023-10-04T00:33:18.860439+0000","last_active":"2023-10-04T01:38:46.315037+0000","last_peered":"2023-10-04T01:38:46.315037+0000","last_clean":"2023-10-04T01:38:46.315037+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:38:46.315037+0000","last_undegraded":"2023-10-04T01:38:46.315037+0000","last_fullsized":"2023-10-04T01:38:46.315037+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:17.831954+0000","last_clean_scrub_stamp":"2023-10-04T00:33:17.831954+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:17:16.511145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":299,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-04T01:37:38.228044+0000","last_change":"2023-10-04T00:33:14.257829+0000","last_active":"2023-10-04T01:37:38.228044+0000","last_peered":"2023-10-04T01:37:38.228044+0000","last_clean":"2023-10-04T01:37:38.228044+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:37:38.228044+0000","last_undegraded":"2023-10-04T01:37:38.228044+0000","last_fullsized":"2023-10-04T01:37:38.228044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:12.807999+0000","last_clean_scrub_stamp":"2023-10-04T00:33:12.807999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T08:35:03.495074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26873135123,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1728000,"num_read_kb":6793160,"num_write":1723504,"num_write_kb":6894010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14124326912,"data_stored":33811423270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40252,"ondisk_log_size":40252,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":54812,"ondisk_log_size":54812,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8514924,"kb_used_data":6900068,"kb_used_omap":2097,"kb_used_meta":1612750,"kb_avail":85856916,"statfs":{"total":96636764160,"available":87917481984,"internally_reserved":0,"allocated":7065669632,"data_stored":16908001535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1651456517},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35899999999999999,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":4.2510000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42299999999999999,"15min":0.45200000000000001},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88100000000000001,"5min":2.145,"15min":4.2869999999999999},"last":0.371}]},{"osd":1,"last update":"Wed Oct 4 01:38:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":2.492,"15min":6.7709999999999999},"last":0.44},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.442,"15min":0.501},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":7.415,"15min":7.415},"last":0.35199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739156,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5239256,"kb_used_data":3392848,"kb_used_omap":997,"kb_used_meta":1845402,"kb_avail":89132584,"statfs":{"total":96636764160,"available":91271766016,"internally_reserved":0,"allocated":3474276352,"data_stored":8324105871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1889692059},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39800000000000002,"15min":0.376},"min":{"1min":0.187,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2810000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.55300000000000005,"15min":0.50800000000000001},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3079999999999998,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.255}]},{"osd":2,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.68400000000000005},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.18099999999999999},"max":{"1min":3.919,"5min":14.467000000000001,"15min":14.467000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.68200000000000005,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71799999999999997,"5min":14.503,"15min":14.503},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359739157,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5290128,"kb_used_data":3505736,"kb_used_omap":1018,"kb_used_meta":1783365,"kb_avail":89081712,"statfs":{"total":96636764160,"available":91219673088,"internally_reserved":0,"allocated":3589873664,"data_stored":8580508134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1826166060},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,6,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.6400000000000001,"15min":1.2090000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.032,"5min":207.047,"15min":207.047},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.37,"5min":2.6899999999999999,"15min":1.2370000000000001},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:38:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.191},"max":{"1min":5.4710000000000001,"5min":5.907,"15min":7.3300000000000001},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3587997696,"data_stored":8579973120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3471486976,"data_stored":8323059731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7064842240,"data_stored":16908390419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:38:53.816 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.816 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.817 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.817 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.817 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.817 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.818 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.818 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.818 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.819 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.819 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.819 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-04T00:33:17.831954+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=17, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.819 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-04T00:33:12.807999+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=12, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:38:53.820 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:39:01.010 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:39:01.002+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:39:01.011 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:39:01.002+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:39:01.011 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:39:01.002+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:39:05.339 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:39:05.340 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:39:05.340 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:39:05.340 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:39:05.347 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.342+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:39:05.347 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.342+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:39:05.347 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.342+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:39:05.347 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:39:05.342+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:39:13.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:39:14.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:39:14.337 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:39:14.381 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2120,"stamp":"2023-10-04T01:39:13.576147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26907415097,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1946596,"num_read_kb":6999645,"num_write":1787292,"num_write_kb":6982707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95304,"ondisk_log_size":95304,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19302924,"kb_used_data":13863092,"kb_used_omap":4113,"kb_used_meta":5435694,"kb_avail":263812596,"statfs":{"total":289910292480,"available":270144098304,"internally_reserved":0,"allocated":14195806208,"data_stored":33878579447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5566150860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,10,5,7,8,12,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16814080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3645,"num_read_kb":14544,"num_write":4073,"num_write_kb":16292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10618,"num_omap_keys":164,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002108"},"pg_stats":[{"pgid":"2.7","version":"141'7509","reported_seq":10659,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:05.738849+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:39:05.738849+0000","last_peered":"2023-10-04T01:39:05.738849+0000","last_clean":"2023-10-04T01:39:05.738849+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:39:05.738849+0000","last_undegraded":"2023-10-04T01:39:05.738849+0000","last_fullsized":"2023-10-04T01:39:05.738849+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7509,"log_dups_size":0,"ondisk_log_size":7509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'7003","reported_seq":10573,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:05.350145+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:39:05.350145+0000","last_peered":"2023-10-04T01:39:05.350145+0000","last_clean":"2023-10-04T01:39:05.350145+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:39:05.350145+0000","last_undegraded":"2023-10-04T01:39:05.350145+0000","last_fullsized":"2023-10-04T01:39:05.350145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7003,"log_dups_size":0,"ondisk_log_size":7003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'7650","reported_seq":11107,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:05.738607+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:39:05.738607+0000","last_peered":"2023-10-04T01:39:05.738607+0000","last_clean":"2023-10-04T01:39:05.738607+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:39:05.738607+0000","last_undegraded":"2023-10-04T01:39:05.738607+0000","last_fullsized":"2023-10-04T01:39:05.738607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'6081","reported_seq":10384,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:05.738650+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:39:05.738650+0000","last_peered":"2023-10-04T01:39:05.738650+0000","last_clean":"2023-10-04T01:39:05.738650+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:39:05.738650+0000","last_undegraded":"2023-10-04T01:39:05.738650+0000","last_fullsized":"2023-10-04T01:39:05.738650+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'441931","reported_seq":872295,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:10.004816+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:39:10.004816+0000","last_peered":"2023-10-04T01:39:10.004816+0000","last_clean":"2023-10-04T01:39:10.004816+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:39:10.004816+0000","last_undegraded":"2023-10-04T01:39:10.004816+0000","last_fullsized":"2023-10-04T01:39:10.004816+0000","mapping_epoch":15,"log_start":"139'431871","ondisk_log_start":"139'431871","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":29,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6756569088,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":435774,"num_read_kb":1713044,"num_write":435280,"num_write_kb":1741120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'15940","reported_seq":20770,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:07.542995+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:39:07.542995+0000","last_peered":"2023-10-04T01:39:07.542995+0000","last_clean":"2023-10-04T01:39:07.542995+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:39:07.542995+0000","last_undegraded":"2023-10-04T01:39:07.542995+0000","last_fullsized":"2023-10-04T01:39:07.542995+0000","mapping_epoch":12,"log_start":"77'5900","ondisk_log_start":"77'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'432366","reported_seq":853490,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:08.767045+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:39:08.767045+0000","last_peered":"2023-10-04T01:39:08.767045+0000","last_clean":"2023-10-04T01:39:08.767045+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:39:08.767045+0000","last_undegraded":"2023-10-04T01:39:08.767045+0000","last_fullsized":"2023-10-04T01:39:08.767045+0000","mapping_epoch":15,"log_start":"139'422360","ondisk_log_start":"139'422360","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":11,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6616379392,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":426435,"num_read_kb":1676444,"num_write":425856,"num_write_kb":1703424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8597,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:10.867033+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:39:10.867033+0000","last_peered":"2023-10-04T01:39:10.867033+0000","last_clean":"2023-10-04T01:39:10.867033+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:39:10.867033+0000","last_undegraded":"2023-10-04T01:39:10.867033+0000","last_fullsized":"2023-10-04T01:39:10.867033+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'451558","reported_seq":892497,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:10.012569+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:39:10.012569+0000","last_peered":"2023-10-04T01:39:10.012569+0000","last_clean":"2023-10-04T01:39:10.012569+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:39:10.012569+0000","last_undegraded":"2023-10-04T01:39:10.012569+0000","last_fullsized":"2023-10-04T01:39:10.012569+0000","mapping_epoch":15,"log_start":"139'441464","ondisk_log_start":"139'441464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":11,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6921764864,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":446490,"num_read_kb":1755980,"num_write":444783,"num_write_kb":1779132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9245,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:10.866729+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:39:10.866729+0000","last_peered":"2023-10-04T01:39:10.866729+0000","last_clean":"2023-10-04T01:39:10.866729+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:39:10.866729+0000","last_undegraded":"2023-10-04T01:39:10.866729+0000","last_fullsized":"2023-10-04T01:39:10.866729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'432241","reported_seq":853642,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:08.765017+0000","last_change":"2023-10-04T01:07:43.400960+0000","last_active":"2023-10-04T01:39:08.765017+0000","last_peered":"2023-10-04T01:39:08.765017+0000","last_clean":"2023-10-04T01:39:08.765017+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:39:08.765017+0000","last_undegraded":"2023-10-04T01:39:08.765017+0000","last_fullsized":"2023-10-04T01:39:08.765017+0000","mapping_epoch":15,"log_start":"139'422163","ondisk_log_start":"139'422163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6612094995,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":426723,"num_read_kb":1677324,"num_write":425729,"num_write_kb":1702910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'16853","reported_seq":21908,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:05.738857+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:39:05.738857+0000","last_peered":"2023-10-04T01:39:05.738857+0000","last_clean":"2023-10-04T01:39:05.738857+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:39:05.738857+0000","last_undegraded":"2023-10-04T01:39:05.738857+0000","last_fullsized":"2023-10-04T01:39:05.738857+0000","mapping_epoch":12,"log_start":"77'6800","ondisk_log_start":"77'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":311,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:54.471038+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:38:54.471038+0000","last_peered":"2023-10-04T01:38:54.471038+0000","last_clean":"2023-10-04T01:38:54.471038+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:38:54.471038+0000","last_undegraded":"2023-10-04T01:38:54.471038+0000","last_fullsized":"2023-10-04T01:38:54.471038+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26906808339,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1735422,"num_read_kb":6822792,"num_write":1731648,"num_write_kb":6926586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14190559232,"data_stored":33877655590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40238,"ondisk_log_size":40238,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":54967,"ondisk_log_size":54967,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8664704,"kb_used_data":6932024,"kb_used_omap":2097,"kb_used_meta":1730574,"kb_avail":85707136,"statfs":{"total":96636764160,"available":87764107264,"internally_reserved":0,"allocated":7098392576,"data_stored":16940716928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1772108293},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35899999999999999,"15min":0.39200000000000002},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.85899999999999999,"5min":2.0680000000000001,"15min":4.2510000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42299999999999999,"15min":0.45200000000000001},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88100000000000001,"5min":2.145,"15min":4.2869999999999999},"last":0.74199999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:38:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":2.492,"15min":6.7709999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.442,"15min":0.501},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":0.57099999999999995,"5min":7.415,"15min":7.415},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5291464,"kb_used_data":3408960,"kb_used_omap":997,"kb_used_meta":1881498,"kb_avail":89080376,"statfs":{"total":96636764160,"available":91218305024,"internally_reserved":0,"allocated":3490775040,"data_stored":8340597008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1926654363},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39800000000000002,"15min":0.376},"min":{"1min":0.187,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2810000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.55300000000000005,"15min":0.50800000000000001},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.3079999999999998,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.34699999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:38:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.79500000000000004,"15min":0.68400000000000005},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.18099999999999999},"max":{"1min":3.919,"5min":14.467000000000001,"15min":14.467000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.68200000000000005,"15min":0.58499999999999996},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71799999999999997,"5min":14.503,"15min":14.503},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739161,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5346756,"kb_used_data":3522108,"kb_used_omap":1018,"kb_used_meta":1823621,"kb_avail":89025084,"statfs":{"total":96636764160,"available":91161686016,"internally_reserved":0,"allocated":3606638592,"data_stored":8597265511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1867388204},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,3,8,12,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.6400000000000001,"15min":1.2090000000000001},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.032,"5min":207.047,"15min":207.047},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.37,"5min":2.6899999999999999,"15min":1.2370000000000001},"min":{"1min":0.23599999999999999,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.308}]},{"osd":2,"last update":"Wed Oct 4 01:38:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.439,"15min":0.47299999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.191},"max":{"1min":5.4710000000000001,"5min":5.907,"15min":7.3300000000000001},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3604738048,"data_stored":8596713472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3488104448,"data_stored":8339677203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7097716736,"data_stored":16941264915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:39:14.383 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:14.383 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:14.383 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:14.384 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:14.384 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:39:34.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:39:34.953 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:39:34.955 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:39:34.980 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2130,"stamp":"2023-10-04T01:39:33.579617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26937459257,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1953435,"num_read_kb":7026925,"num_write":1794586,"num_write_kb":7011883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95455,"ondisk_log_size":95455,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19519272,"kb_used_data":13921168,"kb_used_omap":4113,"kb_used_meta":5593966,"kb_avail":263596248,"statfs":{"total":289910292480,"available":269922557952,"internally_reserved":0,"allocated":14255276032,"data_stored":33938049271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":5728221388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,5,12,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14745600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3332,"num_read_kb":13296,"num_write":3597,"num_write_kb":14388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002076"},"pg_stats":[{"pgid":"2.7","version":"141'7533","reported_seq":10691,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:26.223092+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:39:26.223092+0000","last_peered":"2023-10-04T01:39:26.223092+0000","last_clean":"2023-10-04T01:39:26.223092+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:39:26.223092+0000","last_undegraded":"2023-10-04T01:39:26.223092+0000","last_fullsized":"2023-10-04T01:39:26.223092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7533,"log_dups_size":0,"ondisk_log_size":7533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'7015","reported_seq":10589,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:24.689031+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:39:24.689031+0000","last_peered":"2023-10-04T01:39:24.689031+0000","last_clean":"2023-10-04T01:39:24.689031+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:39:24.689031+0000","last_undegraded":"2023-10-04T01:39:24.689031+0000","last_fullsized":"2023-10-04T01:39:24.689031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'7674","reported_seq":11139,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:26.222622+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:39:26.222622+0000","last_peered":"2023-10-04T01:39:26.222622+0000","last_clean":"2023-10-04T01:39:26.222622+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:39:26.222622+0000","last_undegraded":"2023-10-04T01:39:26.222622+0000","last_fullsized":"2023-10-04T01:39:26.222622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7674,"log_dups_size":0,"ondisk_log_size":7674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'6105","reported_seq":10416,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:26.222658+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:39:26.222658+0000","last_peered":"2023-10-04T01:39:26.222658+0000","last_clean":"2023-10-04T01:39:26.222658+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:39:26.222658+0000","last_undegraded":"2023-10-04T01:39:26.222658+0000","last_fullsized":"2023-10-04T01:39:26.222658+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'443888","reported_seq":875968,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:30.076605+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:39:30.076605+0000","last_peered":"2023-10-04T01:39:30.076605+0000","last_clean":"2023-10-04T01:39:30.076605+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:39:30.076605+0000","last_undegraded":"2023-10-04T01:39:30.076605+0000","last_fullsized":"2023-10-04T01:39:30.076605+0000","mapping_epoch":15,"log_start":"140'433871","ondisk_log_start":"140'433871","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":194,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6764126208,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":437475,"num_read_kb":1719832,"num_write":437105,"num_write_kb":1748420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'15998","reported_seq":20840,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:29.591942+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:39:29.591942+0000","last_peered":"2023-10-04T01:39:29.591942+0000","last_clean":"2023-10-04T01:39:29.591942+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:39:29.591942+0000","last_undegraded":"2023-10-04T01:39:29.591942+0000","last_fullsized":"2023-10-04T01:39:29.591942+0000","mapping_epoch":12,"log_start":"77'5900","ondisk_log_start":"77'5900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'434280","reported_seq":857148,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:28.884873+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:39:28.884873+0000","last_peered":"2023-10-04T01:39:28.884873+0000","last_clean":"2023-10-04T01:39:28.884873+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:39:28.884873+0000","last_undegraded":"2023-10-04T01:39:28.884873+0000","last_fullsized":"2023-10-04T01:39:28.884873+0000","mapping_epoch":15,"log_start":"140'424260","ondisk_log_start":"140'424260","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":179,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 21s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6623748096,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":428168,"num_read_kb":1683360,"num_write":427636,"num_write_kb":1710544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8613,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:31.370716+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:39:31.370716+0000","last_peered":"2023-10-04T01:39:31.370716+0000","last_clean":"2023-10-04T01:39:31.370716+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:39:31.370716+0000","last_undegraded":"2023-10-04T01:39:31.370716+0000","last_fullsized":"2023-10-04T01:39:31.370716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'453553","reported_seq":896289,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:30.071251+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:39:30.071251+0000","last_peered":"2023-10-04T01:39:30.071251+0000","last_clean":"2023-10-04T01:39:30.071251+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:39:30.071251+0000","last_undegraded":"2023-10-04T01:39:30.071251+0000","last_fullsized":"2023-10-04T01:39:30.071251+0000","mapping_epoch":15,"log_start":"140'443464","ondisk_log_start":"140'443464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":149,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 21s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6929485824,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":448279,"num_read_kb":1763116,"num_write":446665,"num_write_kb":1786660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9257,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:31.344338+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:39:31.344338+0000","last_peered":"2023-10-04T01:39:31.344338+0000","last_clean":"2023-10-04T01:39:31.344338+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:39:31.344338+0000","last_undegraded":"2023-10-04T01:39:31.344338+0000","last_fullsized":"2023-10-04T01:39:31.344338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'434048","reported_seq":857070,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:28.904891+0000","last_change":"2023-10-04T01:39:28.270097+0000","last_active":"2023-10-04T01:39:28.904891+0000","last_peered":"2023-10-04T01:39:28.904891+0000","last_clean":"2023-10-04T01:39:28.904891+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:39:28.904891+0000","last_undegraded":"2023-10-04T01:39:28.904891+0000","last_fullsized":"2023-10-04T01:39:28.904891+0000","mapping_epoch":15,"log_start":"140'423963","ondisk_log_start":"140'423963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6619492371,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":428339,"num_read_kb":1683764,"num_write":427536,"num_write_kb":1710138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'16889","reported_seq":21956,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:26.223022+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:39:26.223022+0000","last_peered":"2023-10-04T01:39:26.223022+0000","last_clean":"2023-10-04T01:39:26.223022+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:39:26.223022+0000","last_undegraded":"2023-10-04T01:39:26.223022+0000","last_fullsized":"2023-10-04T01:39:26.223022+0000","mapping_epoch":12,"log_start":"77'6800","ondisk_log_start":"77'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":311,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-04T01:38:54.471038+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:38:54.471038+0000","last_peered":"2023-10-04T01:38:54.471038+0000","last_clean":"2023-10-04T01:38:54.471038+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:38:54.471038+0000","last_undegraded":"2023-10-04T01:38:54.471038+0000","last_fullsized":"2023-10-04T01:38:54.471038+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26936852499,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1742261,"num_read_kb":6850072,"num_write":1738942,"num_write_kb":6955762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14250201088,"data_stored":33937297446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40211,"ondisk_log_size":40211,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55145,"ondisk_log_size":55145,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8761736,"kb_used_data":6961024,"kb_used_omap":2097,"kb_used_meta":1798606,"kb_avail":85610104,"statfs":{"total":96636764160,"available":87664746496,"internally_reserved":0,"allocated":7128088576,"data_stored":16970412928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1841773061},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.4359999999999999,"5min":2.0680000000000001,"15min":3.6749999999999998},"last":0.622},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.442,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.4670000000000001,"5min":2.145,"15min":3.7090000000000001},"last":0.65100000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:39:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40699999999999997,"15min":0.52200000000000002},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.88200000000000001,"5min":2.492,"15min":6.7709999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.83699999999999997,"5min":7.415,"15min":7.415},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359739164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5341748,"kb_used_data":3423148,"kb_used_omap":997,"kb_used_meta":1917594,"kb_avail":89030092,"statfs":{"total":96636764160,"available":91166814208,"internally_reserved":0,"allocated":3505303552,"data_stored":8355125520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1963616667},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.377},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.0900000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.23699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59099999999999997,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72799999999999998,"15min":0.70599999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.18099999999999999},"max":{"1min":6.2439999999999998,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.60299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":6.3520000000000003,"5min":8.1080000000000005,"15min":14.503},"last":0.26600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739165,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5415788,"kb_used_data":3536996,"kb_used_omap":1018,"kb_used_meta":1877765,"kb_avail":88956052,"statfs":{"total":96636764160,"available":91090997248,"internally_reserved":0,"allocated":3621883904,"data_stored":8612510823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1922831660},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:39:12 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":2.653,"15min":1.1910000000000001},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.622,"5min":207.047,"15min":207.047},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.371,"5min":2.6659999999999999,"15min":1.2370000000000001},"min":{"1min":0.246,"5min":0.22800000000000001,"15min":0.20399999999999999},"max":{"1min":0.67600000000000005,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:39:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.621,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45500000000000002,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.9339999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3619934208,"data_stored":8611909632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3502784512,"data_stored":8354357267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7127482368,"data_stored":16971030547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:39:34.983 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:34.983 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:34.983 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:34.984 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:34.984 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:39:54.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:39:55.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:39:55.604 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:39:55.649 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2141,"stamp":"2023-10-04T01:39:55.586077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26975642169,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1962476,"num_read_kb":7062981,"num_write":1803792,"num_write_kb":7048707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95344,"ondisk_log_size":95344,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19912244,"kb_used_data":13989340,"kb_used_omap":4113,"kb_used_meta":5918766,"kb_avail":263203276,"statfs":{"total":289910292480,"available":269520154624,"internally_reserved":0,"allocated":14325084160,"data_stored":34007857399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":6060816588},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1,5,9,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23007232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5518,"num_read_kb":22016,"num_write":5537,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004837"},"pg_stats":[{"pgid":"2.7","version":"141'7541","reported_seq":10709,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.823209+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:39:51.823209+0000","last_peered":"2023-10-04T01:39:51.823209+0000","last_clean":"2023-10-04T01:39:51.823209+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:39:51.823209+0000","last_undegraded":"2023-10-04T01:39:51.823209+0000","last_fullsized":"2023-10-04T01:39:51.823209+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'7019","reported_seq":10598,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:46.703121+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:39:46.703121+0000","last_peered":"2023-10-04T01:39:46.703121+0000","last_clean":"2023-10-04T01:39:46.703121+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:39:46.703121+0000","last_undegraded":"2023-10-04T01:39:46.703121+0000","last_fullsized":"2023-10-04T01:39:46.703121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7019,"log_dups_size":0,"ondisk_log_size":7019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'7682","reported_seq":11157,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.822603+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:39:51.822603+0000","last_peered":"2023-10-04T01:39:51.822603+0000","last_clean":"2023-10-04T01:39:51.822603+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:39:51.822603+0000","last_undegraded":"2023-10-04T01:39:51.822603+0000","last_fullsized":"2023-10-04T01:39:51.822603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'6113","reported_seq":10434,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.822817+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:39:51.822817+0000","last_peered":"2023-10-04T01:39:51.822817+0000","last_clean":"2023-10-04T01:39:51.822817+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:39:51.822817+0000","last_undegraded":"2023-10-04T01:39:51.822817+0000","last_fullsized":"2023-10-04T01:39:51.822817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"141'446367","reported_seq":880755,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:55.292030+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:39:55.292030+0000","last_peered":"2023-10-04T01:39:55.292030+0000","last_clean":"2023-10-04T01:39:55.292030+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:39:55.292030+0000","last_undegraded":"2023-10-04T01:39:55.292030+0000","last_fullsized":"2023-10-04T01:39:55.292030+0000","mapping_epoch":15,"log_start":"140'436272","ondisk_log_start":"140'436272","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":362,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 51s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6773829632,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":439770,"num_read_kb":1728976,"num_write":439446,"num_write_kb":1757784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"141'16049","reported_seq":20906,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:54.751805+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:39:54.751805+0000","last_peered":"2023-10-04T01:39:54.751805+0000","last_clean":"2023-10-04T01:39:54.751805+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:39:54.751805+0000","last_undegraded":"2023-10-04T01:39:54.751805+0000","last_fullsized":"2023-10-04T01:39:54.751805+0000","mapping_epoch":12,"log_start":"77'6000","ondisk_log_start":"77'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"141'436551","reported_seq":861694,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:53.988077+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:39:53.988077+0000","last_peered":"2023-10-04T01:39:53.988077+0000","last_clean":"2023-10-04T01:39:53.988077+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:39:53.988077+0000","last_undegraded":"2023-10-04T01:39:53.988077+0000","last_fullsized":"2023-10-04T01:39:53.988077+0000","mapping_epoch":15,"log_start":"140'426460","ondisk_log_start":"140'426460","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":359,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 46s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6632501248,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":430426,"num_read_kb":1692380,"num_write":429761,"num_write_kb":1719044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8629,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.833237+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:39:51.833237+0000","last_peered":"2023-10-04T01:39:51.833237+0000","last_clean":"2023-10-04T01:39:51.833237+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:39:51.833237+0000","last_undegraded":"2023-10-04T01:39:51.833237+0000","last_fullsized":"2023-10-04T01:39:51.833237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"141'456082","reported_seq":901082,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:39:55.255423+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:39:55.255423+0000","last_peered":"2023-10-04T01:39:55.255423+0000","last_clean":"2023-10-04T01:39:55.255423+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:39:55.255423+0000","last_undegraded":"2023-10-04T01:39:55.255423+0000","last_fullsized":"2023-10-04T01:39:55.255423+0000","mapping_epoch":15,"log_start":"140'446064","ondisk_log_start":"140'446064","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":280,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 47s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6939602944,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":450533,"num_read_kb":1772092,"num_write":449085,"num_write_kb":1796340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9270,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.824228+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:39:51.824228+0000","last_peered":"2023-10-04T01:39:51.824228+0000","last_clean":"2023-10-04T01:39:51.824228+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:39:51.824228+0000","last_undegraded":"2023-10-04T01:39:51.824228+0000","last_fullsized":"2023-10-04T01:39:51.824228+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"141'436368","reported_seq":861633,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:53.990562+0000","last_change":"2023-10-04T01:39:50.100712+0000","last_active":"2023-10-04T01:39:53.990562+0000","last_peered":"2023-10-04T01:39:53.990562+0000","last_clean":"2023-10-04T01:39:53.990562+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:39:53.990562+0000","last_undegraded":"2023-10-04T01:39:53.990562+0000","last_fullsized":"2023-10-04T01:39:53.990562+0000","mapping_epoch":15,"log_start":"140'426363","ondisk_log_start":"140'426363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6629101587,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":430573,"num_read_kb":1692680,"num_write":429856,"num_write_kb":1719418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"141'16901","reported_seq":21983,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:51.823120+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:39:51.823120+0000","last_peered":"2023-10-04T01:39:51.823120+0000","last_clean":"2023-10-04T01:39:51.823120+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:39:51.823120+0000","last_undegraded":"2023-10-04T01:39:51.823120+0000","last_fullsized":"2023-10-04T01:39:51.823120+0000","mapping_epoch":12,"log_start":"77'6900","ondisk_log_start":"77'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":312,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:38:58.105783+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:38:58.105783+0000","last_peered":"2023-10-04T01:38:58.105783+0000","last_clean":"2023-10-04T01:38:58.105783+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:38:58.105783+0000","last_undegraded":"2023-10-04T01:38:58.105783+0000","last_fullsized":"2023-10-04T01:38:58.105783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":26975035411,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1751302,"num_read_kb":6886128,"num_write":1748148,"num_write_kb":6992586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14318518272,"data_stored":34005614630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55036,"ondisk_log_size":55036,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9000816,"kb_used_data":6991912,"kb_used_omap":2097,"kb_used_meta":2006798,"kb_avail":85371024,"statfs":{"total":96636764160,"available":87419928576,"internally_reserved":0,"allocated":7159717888,"data_stored":17002042240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":2054961669},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.4359999999999999,"5min":2.0680000000000001,"15min":3.6749999999999998},"last":0.185},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.442,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.4670000000000001,"5min":2.145,"15min":3.7090000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:39:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40699999999999997,"15min":0.52200000000000002},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.88200000000000001,"5min":2.492,"15min":6.7709999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.83699999999999997,"5min":7.415,"15min":7.415},"last":0.221}]}]},{"osd":1,"up_from":8,"seq":34359739169,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400176,"kb_used_data":3441320,"kb_used_omap":997,"kb_used_meta":1957850,"kb_avail":88971664,"statfs":{"total":96636764160,"available":91106983936,"internally_reserved":0,"allocated":3523911680,"data_stored":8373733648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":2004838811},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.377},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.0900000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.23699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59099999999999997,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.39500000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72799999999999998,"15min":0.70599999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.18099999999999999},"max":{"1min":6.2439999999999998,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.60299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":6.3520000000000003,"5min":8.1080000000000005,"15min":14.503},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739170,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5511252,"kb_used_data":3556108,"kb_used_omap":1018,"kb_used_meta":1954117,"kb_avail":88860588,"statfs":{"total":96636764160,"available":90993242112,"internally_reserved":0,"allocated":3641454592,"data_stored":8632081511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":2001016108},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,11,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:39:12 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":2.653,"15min":1.1910000000000001},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.622,"5min":207.047,"15min":207.047},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.371,"5min":2.6659999999999999,"15min":1.2370000000000001},"min":{"1min":0.246,"5min":0.22800000000000001,"15min":0.20399999999999999},"max":{"1min":0.67600000000000005,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:39:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.621,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":2.2559999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45500000000000002,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.9339999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3639435264,"data_stored":8631410688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3520991232,"data_stored":8372563987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7158091776,"data_stored":17001639955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:39:55.651 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:55.651 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:55.652 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:55.652 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:39:55.652 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:40:02.934 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.928+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:40:02.953 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:40:02.954 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:40:02.955 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:40:02.955 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:02.948+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:40:15.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:40:15.985 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:40:15.985 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:40:15.985 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:40:15.986 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:40:15.986 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:40:15.986 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:40:15.986 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:40:15.986 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:15.980+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:40:16.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:40:16.211 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:40:16.254 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2152,"stamp":"2023-10-04T01:40:15.589382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27004461625,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1969526,"num_read_kb":7091069,"num_write":1810755,"num_write_kb":7076559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95600,"ondisk_log_size":95600,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18095476,"kb_used_data":14043804,"kb_used_omap":4113,"kb_used_meta":4047534,"kb_avail":265020044,"statfs":{"total":289910292480,"available":271380525056,"internally_reserved":0,"allocated":14380855296,"data_stored":34063605753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":4144675020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,10,9,15,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14016512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3285,"num_read_kb":13076,"num_write":3376,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001700"},"pg_stats":[{"pgid":"2.7","version":"142'7577","reported_seq":10754,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.298881+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:40:12.298881+0000","last_peered":"2023-10-04T01:40:12.298881+0000","last_clean":"2023-10-04T01:40:12.298881+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:40:12.298881+0000","last_undegraded":"2023-10-04T01:40:12.298881+0000","last_fullsized":"2023-10-04T01:40:12.298881+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7577,"log_dups_size":0,"ondisk_log_size":7577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'7037","reported_seq":10621,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.095902+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:40:12.095902+0000","last_peered":"2023-10-04T01:40:12.095902+0000","last_clean":"2023-10-04T01:40:12.095902+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:40:12.095902+0000","last_undegraded":"2023-10-04T01:40:12.095902+0000","last_fullsized":"2023-10-04T01:40:12.095902+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7037,"log_dups_size":0,"ondisk_log_size":7037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'7718","reported_seq":11202,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.298896+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:40:12.298896+0000","last_peered":"2023-10-04T01:40:12.298896+0000","last_clean":"2023-10-04T01:40:12.298896+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:40:12.298896+0000","last_undegraded":"2023-10-04T01:40:12.298896+0000","last_fullsized":"2023-10-04T01:40:12.298896+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7718,"log_dups_size":0,"ondisk_log_size":7718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'6149","reported_seq":10479,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.298716+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:40:12.298716+0000","last_peered":"2023-10-04T01:40:12.298716+0000","last_clean":"2023-10-04T01:40:12.298716+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:40:12.298716+0000","last_undegraded":"2023-10-04T01:40:12.298716+0000","last_fullsized":"2023-10-04T01:40:12.298716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'448281","reported_seq":884438,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:15.516259+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:40:15.516259+0000","last_peered":"2023-10-04T01:40:15.516259+0000","last_clean":"2023-10-04T01:40:15.516259+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:40:15.516259+0000","last_undegraded":"2023-10-04T01:40:15.516259+0000","last_fullsized":"2023-10-04T01:40:15.516259+0000","mapping_epoch":15,"log_start":"140'438272","ondisk_log_start":"140'438272","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":519,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 71s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6781186048,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":441532,"num_read_kb":1736000,"num_write":441230,"num_write_kb":1764920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'16094","reported_seq":20964,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:14.943861+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:40:14.943861+0000","last_peered":"2023-10-04T01:40:14.943861+0000","last_clean":"2023-10-04T01:40:14.943861+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:40:14.943861+0000","last_undegraded":"2023-10-04T01:40:14.943861+0000","last_fullsized":"2023-10-04T01:40:14.943861+0000","mapping_epoch":12,"log_start":"77'6000","ondisk_log_start":"77'6000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'438335","reported_seq":865227,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:14.060296+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:40:14.060296+0000","last_peered":"2023-10-04T01:40:14.060296+0000","last_clean":"2023-10-04T01:40:14.060296+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:40:14.060296+0000","last_undegraded":"2023-10-04T01:40:14.060296+0000","last_fullsized":"2023-10-04T01:40:14.060296+0000","mapping_epoch":15,"log_start":"140'428260","ondisk_log_start":"140'428260","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":502,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 66s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6639419392,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":432163,"num_read_kb":1699296,"num_write":431427,"num_write_kb":1725708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8647,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.300608+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:40:12.300608+0000","last_peered":"2023-10-04T01:40:12.300608+0000","last_clean":"2023-10-04T01:40:12.300608+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:40:12.300608+0000","last_undegraded":"2023-10-04T01:40:12.300608+0000","last_fullsized":"2023-10-04T01:40:12.300608+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'457945","reported_seq":904823,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:15.483510+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:40:15.483510+0000","last_peered":"2023-10-04T01:40:15.483510+0000","last_clean":"2023-10-04T01:40:15.483510+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:40:15.483510+0000","last_undegraded":"2023-10-04T01:40:15.483510+0000","last_fullsized":"2023-10-04T01:40:15.483510+0000","mapping_epoch":15,"log_start":"140'447864","ondisk_log_start":"140'447864","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":429,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 67s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6946861056,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":452394,"num_read_kb":1779504,"num_write":450828,"num_write_kb":1803312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9283,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.300176+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:40:12.300176+0000","last_peered":"2023-10-04T01:40:12.300176+0000","last_clean":"2023-10-04T01:40:12.300176+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:40:12.300176+0000","last_undegraded":"2023-10-04T01:40:12.300176+0000","last_fullsized":"2023-10-04T01:40:12.300176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'438138","reported_seq":865100,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:14.065748+0000","last_change":"2023-10-04T01:40:09.245685+0000","last_active":"2023-10-04T01:40:14.065748+0000","last_peered":"2023-10-04T01:40:14.065748+0000","last_clean":"2023-10-04T01:40:14.065748+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:40:14.065748+0000","last_undegraded":"2023-10-04T01:40:14.065748+0000","last_fullsized":"2023-10-04T01:40:14.065748+0000","mapping_epoch":15,"log_start":"140'428063","ondisk_log_start":"140'428063","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6636388371,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":432263,"num_read_kb":1699416,"num_write":431626,"num_write_kb":1726498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'16955","reported_seq":22050,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:12.299520+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:40:12.299520+0000","last_peered":"2023-10-04T01:40:12.299520+0000","last_clean":"2023-10-04T01:40:12.299520+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:40:12.299520+0000","last_undegraded":"2023-10-04T01:40:12.299520+0000","last_fullsized":"2023-10-04T01:40:12.299520+0000","mapping_epoch":12,"log_start":"77'6900","ondisk_log_start":"77'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":313,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:38.944096+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:39:38.944096+0000","last_peered":"2023-10-04T01:39:38.944096+0000","last_clean":"2023-10-04T01:39:38.944096+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:39:38.944096+0000","last_undegraded":"2023-10-04T01:39:38.944096+0000","last_fullsized":"2023-10-04T01:39:38.944096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27003854867,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1758352,"num_read_kb":6914216,"num_write":1755111,"num_write_kb":7020438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14375219200,"data_stored":34062315558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55261,"ondisk_log_size":55261,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739174,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8939200,"kb_used_data":7019192,"kb_used_omap":2097,"kb_used_meta":1917902,"kb_avail":85432640,"statfs":{"total":96636764160,"available":87483023360,"internally_reserved":0,"allocated":7187652608,"data_stored":17029969366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":1963932165},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.4359999999999999,"5min":2.0680000000000001,"15min":3.6749999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.442,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.4670000000000001,"5min":2.145,"15min":3.7090000000000001},"last":0.379}]},{"osd":1,"last update":"Wed Oct 4 01:39:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40699999999999997,"15min":0.52200000000000002},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.88200000000000001,"5min":2.492,"15min":6.7709999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.44900000000000001,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.83699999999999997,"5min":7.415,"15min":7.415},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739173,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4537720,"kb_used_data":3454576,"kb_used_omap":997,"kb_used_meta":1082138,"kb_avail":89834120,"statfs":{"total":96636764160,"available":91990138880,"internally_reserved":0,"allocated":3537485824,"data_stored":8387300198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1108109723},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,5,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.377},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.0900000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.23699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59099999999999997,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.52900000000000003}]},{"osd":2,"last update":"Wed Oct 4 01:39:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72799999999999998,"15min":0.70599999999999996},"min":{"1min":0.192,"5min":0.192,"15min":0.18099999999999999},"max":{"1min":6.2439999999999998,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.60299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":6.3520000000000003,"5min":8.1080000000000005,"15min":14.503},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739174,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4618556,"kb_used_data":3570036,"kb_used_omap":1018,"kb_used_meta":1047493,"kb_avail":89753284,"statfs":{"total":96636764160,"available":91907362816,"internally_reserved":0,"allocated":3655716864,"data_stored":8646336189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1072633132},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,2,9,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":2.6600000000000001,"15min":1.1819999999999999},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.64000000000000001,"5min":207.047,"15min":207.047},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":2.673,"15min":1.2410000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.69599999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:39:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.621,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45500000000000002,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":1.9339999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3653890048,"data_stored":8645865472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3535073280,"data_stored":8386646035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7186255872,"data_stored":17029804051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:40:16.256 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:16.256 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:16.256 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:16.256 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:16.257 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:40:22.706 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:22.700+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:40:22.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:22.700+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:40:22.707 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:22.700+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:40:29.468 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:29.463+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:40:29.468 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:29.463+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:40:29.468 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:29.463+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:40:29.468 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:29.463+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:40:29.468 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:40:29.463+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:40:36.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:40:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:40:36.805 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:40:36.859 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2162,"stamp":"2023-10-04T01:40:35.592795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27030503993,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1976002,"num_read_kb":7116869,"num_write":1817119,"num_write_kb":7102015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95604,"ondisk_log_size":95604,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18272096,"kb_used_data":14098376,"kb_used_omap":4113,"kb_used_meta":4169582,"kb_avail":264843424,"statfs":{"total":289910292480,"available":271199666176,"internally_reserved":0,"allocated":14436737024,"data_stored":34119487481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":4269652172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,4,4,8,6,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18104320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4566,"num_read_kb":18188,"num_write":4436,"num_write_kb":17744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002006"},"pg_stats":[{"pgid":"2.7","version":"142'7614","reported_seq":10799,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:33.110061+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:40:33.110061+0000","last_peered":"2023-10-04T01:40:33.110061+0000","last_clean":"2023-10-04T01:40:33.110061+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:40:33.110061+0000","last_undegraded":"2023-10-04T01:40:33.110061+0000","last_fullsized":"2023-10-04T01:40:33.110061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7614,"log_dups_size":0,"ondisk_log_size":7614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'7055","reported_seq":10643,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:32.782762+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:40:32.782762+0000","last_peered":"2023-10-04T01:40:32.782762+0000","last_clean":"2023-10-04T01:40:32.782762+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:40:32.782762+0000","last_undegraded":"2023-10-04T01:40:32.782762+0000","last_fullsized":"2023-10-04T01:40:32.782762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'7754","reported_seq":11247,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:33.101952+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:40:33.101952+0000","last_peered":"2023-10-04T01:40:33.101952+0000","last_clean":"2023-10-04T01:40:33.101952+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:40:33.101952+0000","last_undegraded":"2023-10-04T01:40:33.101952+0000","last_fullsized":"2023-10-04T01:40:33.101952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'6186","reported_seq":10525,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:33.098662+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:40:33.098662+0000","last_peered":"2023-10-04T01:40:33.098662+0000","last_clean":"2023-10-04T01:40:33.098662+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:40:33.098662+0000","last_undegraded":"2023-10-04T01:40:33.098662+0000","last_fullsized":"2023-10-04T01:40:33.098662+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'449760","reported_seq":887357,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:30.590291+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:40:30.590291+0000","last_peered":"2023-10-04T01:40:30.590291+0000","last_clean":"2023-10-04T01:40:30.590291+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:40:30.590291+0000","last_undegraded":"2023-10-04T01:40:30.590291+0000","last_fullsized":"2023-10-04T01:40:30.590291+0000","mapping_epoch":15,"log_start":"140'439672","ondisk_log_start":"140'439672","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":613,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 86s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6786916352,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":442958,"num_read_kb":1741672,"num_write":442633,"num_write_kb":1770532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'16115","reported_seq":20995,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:30.118995+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:40:30.118995+0000","last_peered":"2023-10-04T01:40:30.118995+0000","last_clean":"2023-10-04T01:40:30.118995+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:40:30.118995+0000","last_undegraded":"2023-10-04T01:40:30.118995+0000","last_fullsized":"2023-10-04T01:40:30.118995+0000","mapping_epoch":12,"log_start":"77'6100","ondisk_log_start":"77'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'440245","reported_seq":868993,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:34.140568+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:40:34.140568+0000","last_peered":"2023-10-04T01:40:34.140568+0000","last_clean":"2023-10-04T01:40:34.140568+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:40:34.140568+0000","last_undegraded":"2023-10-04T01:40:34.140568+0000","last_fullsized":"2023-10-04T01:40:34.140568+0000","mapping_epoch":15,"log_start":"140'430160","ondisk_log_start":"140'430160","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":701,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 86s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6646583296,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":434004,"num_read_kb":1706644,"num_write":433176,"num_write_kb":1732704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8663,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:32.794355+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:40:32.794355+0000","last_peered":"2023-10-04T01:40:32.794355+0000","last_clean":"2023-10-04T01:40:32.794355+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:40:32.794355+0000","last_undegraded":"2023-10-04T01:40:32.794355+0000","last_fullsized":"2023-10-04T01:40:32.794355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'459468","reported_seq":907709,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:30.628911+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:40:30.628911+0000","last_peered":"2023-10-04T01:40:30.628911+0000","last_clean":"2023-10-04T01:40:30.628911+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:40:30.628911+0000","last_undegraded":"2023-10-04T01:40:30.628911+0000","last_fullsized":"2023-10-04T01:40:30.628911+0000","mapping_epoch":15,"log_start":"140'449464","ondisk_log_start":"140'449464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":550,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 82s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6952710144,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":453748,"num_read_kb":1784896,"num_write":452251,"num_write_kb":1809004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9295,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:32.805872+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:40:32.805872+0000","last_peered":"2023-10-04T01:40:32.805872+0000","last_clean":"2023-10-04T01:40:32.805872+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:40:32.805872+0000","last_undegraded":"2023-10-04T01:40:32.805872+0000","last_fullsized":"2023-10-04T01:40:32.805872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'439927","reported_seq":868746,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:34.153878+0000","last_change":"2023-10-04T01:40:26.359989+0000","last_active":"2023-10-04T01:40:34.153878+0000","last_peered":"2023-10-04T01:40:34.153878+0000","last_clean":"2023-10-04T01:40:34.153878+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:40:34.153878+0000","last_undegraded":"2023-10-04T01:40:34.153878+0000","last_fullsized":"2023-10-04T01:40:34.153878+0000","mapping_epoch":15,"log_start":"140'429863","ondisk_log_start":"140'429863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6643687443,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":434118,"num_read_kb":1706804,"num_write":433415,"num_write_kb":1733654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'17009","reported_seq":22117,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:33.178387+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:40:33.178387+0000","last_peered":"2023-10-04T01:40:33.178387+0000","last_clean":"2023-10-04T01:40:33.178387+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:40:33.178387+0000","last_undegraded":"2023-10-04T01:40:33.178387+0000","last_fullsized":"2023-10-04T01:40:33.178387+0000","mapping_epoch":12,"log_start":"77'7000","ondisk_log_start":"77'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":313,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:38.944096+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:39:38.944096+0000","last_peered":"2023-10-04T01:39:38.944096+0000","last_clean":"2023-10-04T01:39:38.944096+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:39:38.944096+0000","last_undegraded":"2023-10-04T01:39:38.944096+0000","last_fullsized":"2023-10-04T01:39:38.944096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27029897235,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1764828,"num_read_kb":6940016,"num_write":1761475,"num_write_kb":7045894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14431252480,"data_stored":34118348838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55264,"ondisk_log_size":55264,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9071044,"kb_used_data":7048316,"kb_used_omap":2097,"kb_used_meta":2020622,"kb_avail":85300796,"statfs":{"total":96636764160,"available":87348015104,"internally_reserved":0,"allocated":7217475584,"data_stored":17059792342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":2069117445},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:39:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.4359999999999999,"5min":2.0680000000000001,"15min":3.6749999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.442,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.4670000000000001,"5min":2.145,"15min":3.7090000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35999999999999999,"15min":0.52300000000000002},"min":{"1min":0.26400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55800000000000005,"5min":0.88200000000000001,"15min":6.7709999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48699999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.68899999999999995,"5min":7.415,"15min":7.415},"last":0.54100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739177,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4557284,"kb_used_data":3468700,"kb_used_omap":997,"kb_used_meta":1087578,"kb_avail":89814556,"statfs":{"total":96636764160,"available":91970105344,"internally_reserved":0,"allocated":3551948800,"data_stored":8401763174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1113680283},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.41999999999999998,"15min":0.36299999999999999},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.221},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54400000000000004,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.25600000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.64600000000000002,"15min":0.68600000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.73899999999999999,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.48799999999999999,"15min":0.58099999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.45500000000000002,"5min":8.1080000000000005,"15min":14.503},"last":0.20599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739177,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4643768,"kb_used_data":3581360,"kb_used_omap":1018,"kb_used_meta":1061381,"kb_avail":89728072,"statfs":{"total":96636764160,"available":91881545728,"internally_reserved":0,"allocated":3667312640,"data_stored":8657931965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1086854444},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,1,4,4,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":2.6600000000000001,"15min":1.1819999999999999},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.64000000000000001,"5min":207.047,"15min":207.047},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":2.673,"15min":1.2410000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.69599999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:40:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.31900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.2559999999999998,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.21199999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.98799999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3665457152,"data_stored":8657432576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3549585408,"data_stored":8401158163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7216209920,"data_stored":17059758099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:40:36.861 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:36.862 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:36.862 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:36.862 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:36.862 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:40:56.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:40:57.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:40:57.379 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:40:57.421 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2172,"stamp":"2023-10-04T01:40:55.596369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27059978809,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1983156,"num_read_kb":7145293,"num_write":1824305,"num_write_kb":7130759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95655,"ondisk_log_size":95655,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18507652,"kb_used_data":14155500,"kb_used_omap":4113,"kb_used_meta":4348014,"kb_avail":264607868,"statfs":{"total":289910292480,"available":270958456832,"internally_reserved":0,"allocated":14495232000,"data_stored":34177982457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4212532,"internal_metadata":4452366540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,5,9,19,18,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18227200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4469,"num_read_kb":17752,"num_write":4449,"num_write_kb":17796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002152"},"pg_stats":[{"pgid":"2.7","version":"142'7632","reported_seq":10826,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:54.068610+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:40:54.068610+0000","last_peered":"2023-10-04T01:40:54.068610+0000","last_clean":"2023-10-04T01:40:54.068610+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:40:54.068610+0000","last_undegraded":"2023-10-04T01:40:54.068610+0000","last_fullsized":"2023-10-04T01:40:54.068610+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'7064","reported_seq":10657,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:53.834931+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:40:53.834931+0000","last_peered":"2023-10-04T01:40:53.834931+0000","last_clean":"2023-10-04T01:40:53.834931+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:40:53.834931+0000","last_undegraded":"2023-10-04T01:40:53.834931+0000","last_fullsized":"2023-10-04T01:40:53.834931+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'7772","reported_seq":11273,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:54.062118+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:40:54.062118+0000","last_peered":"2023-10-04T01:40:54.062118+0000","last_clean":"2023-10-04T01:40:54.062118+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:40:54.062118+0000","last_undegraded":"2023-10-04T01:40:54.062118+0000","last_fullsized":"2023-10-04T01:40:54.062118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'6204","reported_seq":10551,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:54.065420+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:40:54.065420+0000","last_peered":"2023-10-04T01:40:54.065420+0000","last_clean":"2023-10-04T01:40:54.065420+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:40:54.065420+0000","last_undegraded":"2023-10-04T01:40:54.065420+0000","last_fullsized":"2023-10-04T01:40:54.065420+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'451742","reported_seq":891127,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:50.727004+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:40:50.727004+0000","last_peered":"2023-10-04T01:40:50.727004+0000","last_clean":"2023-10-04T01:40:50.727004+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:40:50.727004+0000","last_undegraded":"2023-10-04T01:40:50.727004+0000","last_fullsized":"2023-10-04T01:40:50.727004+0000","mapping_epoch":15,"log_start":"141'441672","ondisk_log_start":"141'441672","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":816,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 106s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6794412032,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":444735,"num_read_kb":1748732,"num_write":444451,"num_write_kb":1777804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'16160","reported_seq":21052,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:49.152571+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:40:49.152571+0000","last_peered":"2023-10-04T01:40:49.152571+0000","last_clean":"2023-10-04T01:40:49.152571+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:40:49.152571+0000","last_undegraded":"2023-10-04T01:40:49.152571+0000","last_fullsized":"2023-10-04T01:40:49.152571+0000","mapping_epoch":12,"log_start":"77'6100","ondisk_log_start":"77'6100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'442101","reported_seq":872668,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:54.302332+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:40:54.302332+0000","last_peered":"2023-10-04T01:40:54.302332+0000","last_clean":"2023-10-04T01:40:54.302332+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:40:54.302332+0000","last_undegraded":"2023-10-04T01:40:54.302332+0000","last_fullsized":"2023-10-04T01:40:54.302332+0000","mapping_epoch":15,"log_start":"141'432061","ondisk_log_start":"141'432061","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":874,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 106s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6653591552,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":435813,"num_read_kb":1713824,"num_write":434893,"num_write_kb":1739572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8675,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:48.166394+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:40:48.166394+0000","last_peered":"2023-10-04T01:40:48.166394+0000","last_clean":"2023-10-04T01:40:48.166394+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:40:48.166394+0000","last_undegraded":"2023-10-04T01:40:48.166394+0000","last_fullsized":"2023-10-04T01:40:48.166394+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'461507","reported_seq":911568,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:40:50.739646+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:40:50.739646+0000","last_peered":"2023-10-04T01:40:50.739646+0000","last_clean":"2023-10-04T01:40:50.739646+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:40:50.739646+0000","last_undegraded":"2023-10-04T01:40:50.739646+0000","last_fullsized":"2023-10-04T01:40:50.739646+0000","mapping_epoch":15,"log_start":"141'451464","ondisk_log_start":"141'451464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":706,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6960541696,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":455559,"num_read_kb":1792092,"num_write":454162,"num_write_kb":1816648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9304,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:48.160033+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:40:48.160033+0000","last_peered":"2023-10-04T01:40:48.160033+0000","last_clean":"2023-10-04T01:40:48.160033+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:40:48.160033+0000","last_undegraded":"2023-10-04T01:40:48.160033+0000","last_fullsized":"2023-10-04T01:40:48.160033+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'441667","reported_seq":872251,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:54.300210+0000","last_change":"2023-10-04T01:40:52.624260+0000","last_active":"2023-10-04T01:40:54.300210+0000","last_peered":"2023-10-04T01:40:54.300210+0000","last_clean":"2023-10-04T01:40:54.300210+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:40:54.300210+0000","last_undegraded":"2023-10-04T01:40:54.300210+0000","last_fullsized":"2023-10-04T01:40:54.300210+0000","mapping_epoch":15,"log_start":"141'431663","ondisk_log_start":"141'431663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6650826771,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":435875,"num_read_kb":1713792,"num_write":435155,"num_write_kb":1740614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'17036","reported_seq":22156,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:40:54.090432+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:40:54.090432+0000","last_peered":"2023-10-04T01:40:54.090432+0000","last_clean":"2023-10-04T01:40:54.090432+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:40:54.090432+0000","last_undegraded":"2023-10-04T01:40:54.090432+0000","last_fullsized":"2023-10-04T01:40:54.090432+0000","mapping_epoch":12,"log_start":"77'7000","ondisk_log_start":"77'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":313,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:38.944096+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:39:38.944096+0000","last_peered":"2023-10-04T01:39:38.944096+0000","last_clean":"2023-10-04T01:39:38.944096+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:39:38.944096+0000","last_undegraded":"2023-10-04T01:39:38.944096+0000","last_fullsized":"2023-10-04T01:39:38.944096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27059372051,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1771982,"num_read_kb":6968440,"num_write":1768661,"num_write_kb":7074638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14489939968,"data_stored":34177036326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40157,"ondisk_log_size":40157,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55399,"ondisk_log_size":55399,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739182,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9167412,"kb_used_data":7076652,"kb_used_omap":2097,"kb_used_meta":2088654,"kb_avail":85204428,"statfs":{"total":96636764160,"available":87249334272,"internally_reserved":0,"allocated":7246491648,"data_stored":17088808406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2147835,"internal_metadata":2138782213},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:40:33 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.35199999999999998,"15min":0.36699999999999999},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.5,"5min":1.4359999999999999,"15min":2.9550000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.40999999999999998,"15min":0.42799999999999999},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53700000000000003,"5min":1.4670000000000001,"15min":2.9900000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35999999999999999,"15min":0.52300000000000002},"min":{"1min":0.26400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55800000000000005,"5min":0.88200000000000001,"15min":6.7709999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48699999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.68899999999999995,"5min":7.415,"15min":7.415},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739181,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4609368,"kb_used_data":3482576,"kb_used_omap":997,"kb_used_meta":1125786,"kb_avail":89762472,"statfs":{"total":96636764160,"available":91916771328,"internally_reserved":0,"allocated":3566157824,"data_stored":8415972198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1152805275},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.41999999999999998,"15min":0.36299999999999999},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54400000000000004,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.64600000000000002,"15min":0.68600000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.73899999999999999,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.504},{"interface":"front","average":{"1min":0.316,"5min":0.48799999999999999,"15min":0.58099999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.45500000000000002,"5min":8.1080000000000005,"15min":14.503},"last":0.46800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4730872,"kb_used_data":3596272,"kb_used_omap":1018,"kb_used_meta":1133573,"kb_avail":89640968,"statfs":{"total":96636764160,"available":91792351232,"internally_reserved":0,"allocated":3682582528,"data_stored":8673201853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1160779052},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,7,16,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":2.6600000000000001,"15min":1.1819999999999999},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.64000000000000001,"5min":207.047,"15min":207.047},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":2.673,"15min":1.2410000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.69599999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:40:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.31900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.2559999999999998,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.21199999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.98799999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3680718848,"data_stored":8672694272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3563716608,"data_stored":8415289363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7245504512,"data_stored":17089052691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:40:57.424 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:57.424 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:57.424 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:57.424 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:40:57.425 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:41:17.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:41:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:41:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:41:17.985 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2183,"stamp":"2023-10-04T01:41:17.600199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27093512761,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1991309,"num_read_kb":7177749,"num_write":1832488,"num_write_kb":7163491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95792,"ondisk_log_size":95792,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18046748,"kb_used_data":14216644,"kb_used_omap":4115,"kb_used_meta":3825964,"kb_avail":265068772,"statfs":{"total":289910292480,"available":271430422528,"internally_reserved":0,"allocated":14557843456,"data_stored":34240593913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":3917787705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,0,5,8,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18632704,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4561,"num_read_kb":18160,"num_write":4543,"num_write_kb":18172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002047"},"pg_stats":[{"pgid":"2.7","version":"142'7656","reported_seq":10858,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:13.742999+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:41:13.742999+0000","last_peered":"2023-10-04T01:41:13.742999+0000","last_clean":"2023-10-04T01:41:13.742999+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:41:13.742999+0000","last_undegraded":"2023-10-04T01:41:13.742999+0000","last_fullsized":"2023-10-04T01:41:13.742999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7656,"log_dups_size":0,"ondisk_log_size":7656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'7076","reported_seq":10673,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:12.974058+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:41:12.974058+0000","last_peered":"2023-10-04T01:41:12.974058+0000","last_clean":"2023-10-04T01:41:12.974058+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:41:12.974058+0000","last_undegraded":"2023-10-04T01:41:12.974058+0000","last_fullsized":"2023-10-04T01:41:12.974058+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'7796","reported_seq":11305,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:13.742705+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:41:13.742705+0000","last_peered":"2023-10-04T01:41:13.742705+0000","last_clean":"2023-10-04T01:41:13.742705+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:41:13.742705+0000","last_undegraded":"2023-10-04T01:41:13.742705+0000","last_fullsized":"2023-10-04T01:41:13.742705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'6228","reported_seq":10583,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:13.743081+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:41:13.743081+0000","last_peered":"2023-10-04T01:41:13.743081+0000","last_clean":"2023-10-04T01:41:13.743081+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:41:13.743081+0000","last_undegraded":"2023-10-04T01:41:13.743081+0000","last_fullsized":"2023-10-04T01:41:13.743081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6228,"log_dups_size":0,"ondisk_log_size":6228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"142'454208","reported_seq":895931,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:15.876615+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:41:15.876615+0000","last_peered":"2023-10-04T01:41:15.876615+0000","last_clean":"2023-10-04T01:41:15.876615+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:41:15.876615+0000","last_undegraded":"2023-10-04T01:41:15.876615+0000","last_fullsized":"2023-10-04T01:41:15.876615+0000","mapping_epoch":15,"log_start":"141'444172","ondisk_log_start":"141'444172","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":961,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 131s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6804037632,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":447061,"num_read_kb":1757968,"num_write":446802,"num_write_kb":1787208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"142'16214","reported_seq":21121,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:14.660130+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:41:14.660130+0000","last_peered":"2023-10-04T01:41:14.660130+0000","last_clean":"2023-10-04T01:41:14.660130+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:41:14.660130+0000","last_undegraded":"2023-10-04T01:41:14.660130+0000","last_fullsized":"2023-10-04T01:41:14.660130+0000","mapping_epoch":12,"log_start":"77'6200","ondisk_log_start":"77'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"142'444032","reported_seq":876383,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:14.368250+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:41:14.368250+0000","last_peered":"2023-10-04T01:41:14.368250+0000","last_clean":"2023-10-04T01:41:14.368250+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:41:14.368250+0000","last_undegraded":"2023-10-04T01:41:14.368250+0000","last_fullsized":"2023-10-04T01:41:14.368250+0000","mapping_epoch":15,"log_start":"141'433961","ondisk_log_start":"141'433961","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1050,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 127s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6660960256,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":437581,"num_read_kb":1720868,"num_write":436684,"num_write_kb":1746736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8691,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:08.619812+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:41:08.619812+0000","last_peered":"2023-10-04T01:41:08.619812+0000","last_clean":"2023-10-04T01:41:08.619812+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:41:08.619812+0000","last_undegraded":"2023-10-04T01:41:08.619812+0000","last_fullsized":"2023-10-04T01:41:08.619812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"142'464004","reported_seq":916415,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:15.854804+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:41:15.854804+0000","last_peered":"2023-10-04T01:41:15.854804+0000","last_clean":"2023-10-04T01:41:15.854804+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:41:15.854804+0000","last_undegraded":"2023-10-04T01:41:15.854804+0000","last_fullsized":"2023-10-04T01:41:15.854804+0000","mapping_epoch":15,"log_start":"141'453964","ondisk_log_start":"141'453964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":865,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 127s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6970261504,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":457893,"num_read_kb":1801396,"num_write":456534,"num_write_kb":1826136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9316,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:08.626326+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:41:08.626326+0000","last_peered":"2023-10-04T01:41:08.626326+0000","last_clean":"2023-10-04T01:41:08.626326+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:41:08.626326+0000","last_undegraded":"2023-10-04T01:41:08.626326+0000","last_fullsized":"2023-10-04T01:41:08.626326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"142'443336","reported_seq":875649,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:14.351004+0000","last_change":"2023-10-04T01:41:05.681963+0000","last_active":"2023-10-04T01:41:14.351004+0000","last_peered":"2023-10-04T01:41:14.351004+0000","last_clean":"2023-10-04T01:41:14.351004+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:41:14.351004+0000","last_undegraded":"2023-10-04T01:41:14.351004+0000","last_fullsized":"2023-10-04T01:41:14.351004+0000","mapping_epoch":15,"log_start":"141'433263","ondisk_log_start":"141'433263","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6657646611,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":437600,"num_read_kb":1720664,"num_write":436824,"num_write_kb":1747290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"142'17072","reported_seq":22204,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-04T01:41:13.743690+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:41:13.743690+0000","last_peered":"2023-10-04T01:41:13.743690+0000","last_clean":"2023-10-04T01:41:13.743690+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:41:13.743690+0000","last_undegraded":"2023-10-04T01:41:13.743690+0000","last_fullsized":"2023-10-04T01:41:13.743690+0000","mapping_epoch":12,"log_start":"77'7000","ondisk_log_start":"77'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":313,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-04T01:39:38.944096+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:39:38.944096+0000","last_peered":"2023-10-04T01:39:38.944096+0000","last_clean":"2023-10-04T01:39:38.944096+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:39:38.944096+0000","last_undegraded":"2023-10-04T01:39:38.944096+0000","last_fullsized":"2023-10-04T01:39:38.944096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27092906003,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1780135,"num_read_kb":7000896,"num_write":1776844,"num_write_kb":7107370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14552907776,"data_stored":34240004134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40220,"ondisk_log_size":40220,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55473,"ondisk_log_size":55473,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739186,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8566292,"kb_used_data":7105036,"kb_used_omap":2099,"kb_used_meta":1459148,"kb_avail":85805548,"statfs":{"total":96636764160,"available":87864881152,"internally_reserved":0,"allocated":7275556864,"data_stored":17117873622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1494168434},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:40:33 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.35199999999999998,"15min":0.36699999999999999},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.5,"5min":1.4359999999999999,"15min":2.9550000000000001},"last":0.31},{"interface":"front","average":{"1min":0.372,"5min":0.40999999999999998,"15min":0.42799999999999999},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53700000000000003,"5min":1.4670000000000001,"15min":2.9900000000000002},"last":0.34300000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35999999999999999,"15min":0.52300000000000002},"min":{"1min":0.26400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55800000000000005,"5min":0.88200000000000001,"15min":6.7709999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.42299999999999999,"15min":0.48699999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.20000000000000001},"max":{"1min":0.68899999999999995,"5min":7.415,"15min":7.415},"last":0.19600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4659344,"kb_used_data":3496456,"kb_used_omap":997,"kb_used_meta":1161882,"kb_avail":89712496,"statfs":{"total":96636764160,"available":91865595904,"internally_reserved":0,"allocated":3580370944,"data_stored":8430185318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1189767579},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.41999999999999998,"15min":0.36299999999999999},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":2.8050000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54400000000000004,"5min":5.6520000000000001,"15min":6.1820000000000004},"last":0.27200000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:40:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.64600000000000002,"15min":0.68600000000000005},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.73899999999999999,"5min":8.0709999999999997,"15min":14.467000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.48799999999999999,"15min":0.58099999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.45500000000000002,"5min":8.1080000000000005,"15min":14.503},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739186,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4821112,"kb_used_data":3615152,"kb_used_omap":1018,"kb_used_meta":1204933,"kb_avail":89550728,"statfs":{"total":96636764160,"available":91699945472,"internally_reserved":0,"allocated":3701915648,"data_stored":8692534973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1233851692},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,0,3,7,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:40:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":2.6600000000000001,"15min":1.1819999999999999},"min":{"1min":0.20599999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.64000000000000001,"5min":207.047,"15min":207.047},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":2.673,"15min":1.2410000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.69599999999999995,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.312}]},{"osd":2,"last update":"Wed Oct 4 01:40:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.31900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.2559999999999998,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.45900000000000002,"15min":0.54300000000000004},"min":{"1min":0.21199999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":0.98799999999999999,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3700072448,"data_stored":8692047872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577892864,"data_stored":8429465619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7274942464,"data_stored":17118490643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:41:17.987 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:17.988 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:17.988 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:17.988 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:17.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:41:18.531 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:41:18.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:41:19.128 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:41:19.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:41:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:41:19.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:41:20.284 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:41:20.345 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:41:21.107 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:41:21.098+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:41:21.108 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:41:21.098+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:41:21.108 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:41:21.098+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:41:40.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:41:40.858 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:41:40.859 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:41:40.897 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2195,"stamp":"2023-10-04T01:41:39.606958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27127095865,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":1999598,"num_read_kb":7210773,"num_write":1840691,"num_write_kb":7196303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95922,"ondisk_log_size":95922,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18328956,"kb_used_data":14285796,"kb_used_omap":4115,"kb_used_meta":4039020,"kb_avail":264786564,"statfs":{"total":289910292480,"available":271141441536,"internally_reserved":0,"allocated":14628655104,"data_stored":34311382650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4135957049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,4,3,10,14,17,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18747392,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4653,"num_read_kb":18556,"num_write":4574,"num_write_kb":18296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002122"},"pg_stats":[{"pgid":"2.7","version":"143'7690","reported_seq":10903,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.338954+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:41:39.338954+0000","last_peered":"2023-10-04T01:41:39.338954+0000","last_clean":"2023-10-04T01:41:39.338954+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:41:39.338954+0000","last_undegraded":"2023-10-04T01:41:39.338954+0000","last_fullsized":"2023-10-04T01:41:39.338954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'7093","reported_seq":10696,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:38.433558+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:41:38.433558+0000","last_peered":"2023-10-04T01:41:38.433558+0000","last_clean":"2023-10-04T01:41:38.433558+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:41:38.433558+0000","last_undegraded":"2023-10-04T01:41:38.433558+0000","last_fullsized":"2023-10-04T01:41:38.433558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7093,"log_dups_size":0,"ondisk_log_size":7093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'7830","reported_seq":11350,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.338620+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:41:39.338620+0000","last_peered":"2023-10-04T01:41:39.338620+0000","last_clean":"2023-10-04T01:41:39.338620+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:41:39.338620+0000","last_undegraded":"2023-10-04T01:41:39.338620+0000","last_fullsized":"2023-10-04T01:41:39.338620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'6262","reported_seq":10628,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.338873+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:41:39.338873+0000","last_peered":"2023-10-04T01:41:39.338873+0000","last_clean":"2023-10-04T01:41:39.338873+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:41:39.338873+0000","last_undegraded":"2023-10-04T01:41:39.338873+0000","last_fullsized":"2023-10-04T01:41:39.338873+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'456231","reported_seq":899831,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:36.267048+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:41:36.267048+0000","last_peered":"2023-10-04T01:41:36.267048+0000","last_clean":"2023-10-04T01:41:36.267048+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:41:36.267048+0000","last_undegraded":"2023-10-04T01:41:36.267048+0000","last_fullsized":"2023-10-04T01:41:36.267048+0000","mapping_epoch":15,"log_start":"141'446172","ondisk_log_start":"141'446172","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1156,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6811680768,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":448925,"num_read_kb":1765388,"num_write":448669,"num_write_kb":1794676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'16241","reported_seq":21161,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:34.218777+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:41:34.218777+0000","last_peered":"2023-10-04T01:41:34.218777+0000","last_clean":"2023-10-04T01:41:34.218777+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:41:34.218777+0000","last_undegraded":"2023-10-04T01:41:34.218777+0000","last_fullsized":"2023-10-04T01:41:34.218777+0000","mapping_epoch":12,"log_start":"77'6200","ondisk_log_start":"77'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'446381","reported_seq":881025,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:39.530812+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:41:39.530812+0000","last_peered":"2023-10-04T01:41:39.530812+0000","last_clean":"2023-10-04T01:41:39.530812+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:41:39.530812+0000","last_undegraded":"2023-10-04T01:41:39.530812+0000","last_fullsized":"2023-10-04T01:41:39.530812+0000","mapping_epoch":15,"log_start":"141'436361","ondisk_log_start":"141'436361","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1236,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 152s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6669950976,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":439859,"num_read_kb":1729932,"num_write":438880,"num_write_kb":1755520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8712,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:34.228374+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:41:34.228374+0000","last_peered":"2023-10-04T01:41:34.228374+0000","last_clean":"2023-10-04T01:41:34.228374+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:41:34.228374+0000","last_undegraded":"2023-10-04T01:41:34.228374+0000","last_fullsized":"2023-10-04T01:41:34.228374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'466047","reported_seq":920373,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:36.266243+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:41:36.266243+0000","last_peered":"2023-10-04T01:41:36.266243+0000","last_clean":"2023-10-04T01:41:36.266243+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:41:36.266243+0000","last_undegraded":"2023-10-04T01:41:36.266243+0000","last_fullsized":"2023-10-04T01:41:36.266243+0000","mapping_epoch":15,"log_start":"141'455964","ondisk_log_start":"141'455964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1015,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 148s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6978121728,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":459798,"num_read_kb":1809000,"num_write":458456,"num_write_kb":1833824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9332,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:34.219377+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:41:34.219377+0000","last_peered":"2023-10-04T01:41:34.219377+0000","last_clean":"2023-10-04T01:41:34.219377+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:41:34.219377+0000","last_undegraded":"2023-10-04T01:41:34.219377+0000","last_fullsized":"2023-10-04T01:41:34.219377+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'445554","reported_seq":880121,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.535651+0000","last_change":"2023-10-04T01:41:36.779905+0000","last_active":"2023-10-04T01:41:39.535651+0000","last_peered":"2023-10-04T01:41:39.535651+0000","last_clean":"2023-10-04T01:41:39.535651+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:41:39.535651+0000","last_undegraded":"2023-10-04T01:41:39.535651+0000","last_fullsized":"2023-10-04T01:41:39.535651+0000","mapping_epoch":15,"log_start":"141'435463","ondisk_log_start":"141'435463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6666735635,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":439842,"num_read_kb":1729600,"num_write":439042,"num_write_kb":1756162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'17123","reported_seq":22271,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.338947+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:41:39.338947+0000","last_peered":"2023-10-04T01:41:39.338947+0000","last_clean":"2023-10-04T01:41:39.338947+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:41:39.338947+0000","last_undegraded":"2023-10-04T01:41:39.338947+0000","last_fullsized":"2023-10-04T01:41:39.338947+0000","mapping_epoch":12,"log_start":"77'7100","ondisk_log_start":"77'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":315,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:18.156432+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:18.156432+0000","last_peered":"2023-10-04T01:41:18.156432+0000","last_clean":"2023-10-04T01:41:18.156432+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:18.156432+0000","last_undegraded":"2023-10-04T01:41:18.156432+0000","last_fullsized":"2023-10-04T01:41:18.156432+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27126489107,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1788424,"num_read_kb":7033920,"num_write":1785047,"num_write_kb":7140182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14623707136,"data_stored":34310803494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40253,"ondisk_log_size":40253,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55570,"ondisk_log_size":55570,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739191,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8733312,"kb_used_data":7141560,"kb_used_omap":2099,"kb_used_meta":1589644,"kb_avail":85638528,"statfs":{"total":96636764160,"available":87693852672,"internally_reserved":0,"allocated":7312957440,"data_stored":17155266561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1627796338},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.41099999999999998,"15min":0.38300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.2779999999999996,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.437},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.443},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":5.319,"5min":5.319,"15min":5.319},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Oct 4 01:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.51500000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.657,"5min":1.657,"15min":6.7709999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.9509999999999996,"5min":7.415,"15min":7.415},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739190,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4741256,"kb_used_data":3513856,"kb_used_omap":997,"kb_used_meta":1226394,"kb_avail":89630584,"statfs":{"total":96636764160,"available":91781718016,"internally_reserved":0,"allocated":3598188544,"data_stored":8447995281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1255827867},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,1,2,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.373},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":5.7240000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38700000000000001,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67400000000000004,"5min":2.3079999999999998,"15min":6.1820000000000004},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.65100000000000002,"15min":0.69899999999999995},"min":{"1min":0.21099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.3469999999999995,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.58099999999999996},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.2489999999999997,"5min":8.1080000000000005,"15min":14.503},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4854388,"kb_used_data":3630380,"kb_used_omap":1018,"kb_used_meta":1222981,"kb_avail":89517452,"statfs":{"total":96636764160,"available":91665870848,"internally_reserved":0,"allocated":3717509120,"data_stored":8708120808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1252332844},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,8,5,17,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":2.673,"15min":1.153},"min":{"1min":0.24199999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.899,"5min":207.047,"15min":207.047},"last":0.19},{"interface":"front","average":{"1min":0.58099999999999996,"5min":2.7170000000000001,"15min":1.216},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":2.6480000000000001,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:41:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.53800000000000003,"15min":0.48599999999999999},"min":{"1min":0.27200000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.0999999999999996,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.315},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.0019999999999998,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3715584000,"data_stored":8707559424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3595964416,"data_stored":8447537171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7312158720,"data_stored":17155706899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:41:40.899 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:40.899 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:40.899 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:40.900 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:41:40.900 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:41:48.455 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:48.449+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:41:51.784 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.777+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:41:51.784 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.777+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:41:51.785 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.777+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:41:51.787 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.785+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:41:51.794 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.789+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:41:51.794 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.789+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:41:51.795 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.789+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:41:51.795 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.789+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:41:51.795 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:41:51.789+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:42:00.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:42:01.540 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:42:01.541 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:42:01.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2205,"stamp":"2023-10-04T01:41:59.612302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27153523257,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2005905,"num_read_kb":7235893,"num_write":1847138,"num_write_kb":7222091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96006,"ondisk_log_size":96006,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18583064,"kb_used_data":14340800,"kb_used_omap":4115,"kb_used_meta":4238124,"kb_avail":264532456,"statfs":{"total":289910292480,"available":270881234944,"internally_reserved":0,"allocated":14684979200,"data_stored":34367706746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4339839545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,7,5,3,16,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15020032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3564,"num_read_kb":14184,"num_write":3659,"num_write_kb":14636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002048"},"pg_stats":[{"pgid":"2.7","version":"143'7710","reported_seq":10927,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.212597+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:41:54.212597+0000","last_peered":"2023-10-04T01:41:54.212597+0000","last_clean":"2023-10-04T01:41:54.212597+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:41:54.212597+0000","last_undegraded":"2023-10-04T01:41:54.212597+0000","last_fullsized":"2023-10-04T01:41:54.212597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'7104","reported_seq":10709,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:52.706312+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:41:52.706312+0000","last_peered":"2023-10-04T01:41:52.706312+0000","last_clean":"2023-10-04T01:41:52.706312+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:41:52.706312+0000","last_undegraded":"2023-10-04T01:41:52.706312+0000","last_fullsized":"2023-10-04T01:41:52.706312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7104,"log_dups_size":0,"ondisk_log_size":7104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'7851","reported_seq":11375,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.217852+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:41:54.217852+0000","last_peered":"2023-10-04T01:41:54.217852+0000","last_clean":"2023-10-04T01:41:54.217852+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:41:54.217852+0000","last_undegraded":"2023-10-04T01:41:54.217852+0000","last_fullsized":"2023-10-04T01:41:54.217852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'6282","reported_seq":10652,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.220452+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:41:54.220452+0000","last_peered":"2023-10-04T01:41:54.220452+0000","last_clean":"2023-10-04T01:41:54.220452+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:41:54.220452+0000","last_undegraded":"2023-10-04T01:41:54.220452+0000","last_fullsized":"2023-10-04T01:41:54.220452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'458196","reported_seq":903589,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:56.383321+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:41:56.383321+0000","last_peered":"2023-10-04T01:41:56.383321+0000","last_clean":"2023-10-04T01:41:56.383321+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:41:56.383321+0000","last_undegraded":"2023-10-04T01:41:56.383321+0000","last_fullsized":"2023-10-04T01:41:56.383321+0000","mapping_epoch":15,"log_start":"142'448172","ondisk_log_start":"142'448172","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1276,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 172s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6819340288,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":450702,"num_read_kb":1772468,"num_write":450540,"num_write_kb":1802160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'16271","reported_seq":21203,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.699067+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:41:54.699067+0000","last_peered":"2023-10-04T01:41:54.699067+0000","last_clean":"2023-10-04T01:41:54.699067+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:41:54.699067+0000","last_undegraded":"2023-10-04T01:41:54.699067+0000","last_fullsized":"2023-10-04T01:41:54.699067+0000","mapping_epoch":12,"log_start":"77'6200","ondisk_log_start":"77'6200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'447801","reported_seq":883746,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:54.621080+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:41:54.621080+0000","last_peered":"2023-10-04T01:41:54.621080+0000","last_clean":"2023-10-04T01:41:54.621080+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:41:54.621080+0000","last_undegraded":"2023-10-04T01:41:54.621080+0000","last_fullsized":"2023-10-04T01:41:54.621080+0000","mapping_epoch":15,"log_start":"141'437762","ondisk_log_start":"141'437762","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1350,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 167s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6675382272,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":441148,"num_read_kb":1735072,"num_write":440207,"num_write_kb":1760828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8728,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.731661+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:41:54.731661+0000","last_peered":"2023-10-04T01:41:54.731661+0000","last_clean":"2023-10-04T01:41:54.731661+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:41:54.731661+0000","last_undegraded":"2023-10-04T01:41:54.731661+0000","last_fullsized":"2023-10-04T01:41:54.731661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'468012","reported_seq":924231,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:41:56.389277+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:41:56.389277+0000","last_peered":"2023-10-04T01:41:56.389277+0000","last_clean":"2023-10-04T01:41:56.389277+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:41:56.389277+0000","last_undegraded":"2023-10-04T01:41:56.389277+0000","last_fullsized":"2023-10-04T01:41:56.389277+0000","mapping_epoch":15,"log_start":"142'457964","ondisk_log_start":"142'457964","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1162,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6985666560,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":461674,"num_read_kb":1816472,"num_write":460303,"num_write_kb":1841212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9345,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.810326+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:41:54.810326+0000","last_peered":"2023-10-04T01:41:54.810326+0000","last_clean":"2023-10-04T01:41:54.810326+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:41:54.810326+0000","last_undegraded":"2023-10-04T01:41:54.810326+0000","last_fullsized":"2023-10-04T01:41:54.810326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'446956","reported_seq":882897,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.620137+0000","last_change":"2023-10-04T01:41:50.793268+0000","last_active":"2023-10-04T01:41:54.620137+0000","last_peered":"2023-10-04T01:41:54.620137+0000","last_clean":"2023-10-04T01:41:54.620137+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:41:54.620137+0000","last_undegraded":"2023-10-04T01:41:54.620137+0000","last_fullsized":"2023-10-04T01:41:54.620137+0000","mapping_epoch":15,"log_start":"141'436863","ondisk_log_start":"141'436863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6672527379,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":441207,"num_read_kb":1735028,"num_write":440444,"num_write_kb":1761770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'17154","reported_seq":22308,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:54.275380+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:41:54.275380+0000","last_peered":"2023-10-04T01:41:54.275380+0000","last_clean":"2023-10-04T01:41:54.275380+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:41:54.275380+0000","last_undegraded":"2023-10-04T01:41:54.275380+0000","last_fullsized":"2023-10-04T01:41:54.275380+0000","mapping_epoch":12,"log_start":"77'7100","ondisk_log_start":"77'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":315,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:18.156432+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:18.156432+0000","last_peered":"2023-10-04T01:41:18.156432+0000","last_clean":"2023-10-04T01:41:18.156432+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:18.156432+0000","last_undegraded":"2023-10-04T01:41:18.156432+0000","last_fullsized":"2023-10-04T01:41:18.156432+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27152916499,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1794731,"num_read_kb":7059040,"num_write":1791494,"num_write_kb":7165970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14680100864,"data_stored":34367197222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40204,"ondisk_log_size":40204,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55703,"ondisk_log_size":55703,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8868924,"kb_used_data":7170740,"kb_used_omap":2099,"kb_used_meta":1696076,"kb_avail":85502916,"statfs":{"total":96636764160,"available":87554985984,"internally_reserved":0,"allocated":7342837760,"data_stored":17185146881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1736782706},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.41099999999999998,"15min":0.38300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.2779999999999996,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.314},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.443},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":5.319,"5min":5.319,"15min":5.319},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.51500000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.657,"5min":1.657,"15min":6.7709999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.9509999999999996,"5min":7.415,"15min":7.415},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4766156,"kb_used_data":3524868,"kb_used_omap":997,"kb_used_meta":1240282,"kb_avail":89605684,"statfs":{"total":96636764160,"available":91756220416,"internally_reserved":0,"allocated":3609464832,"data_stored":8459271569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1270049179},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.373},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":5.7240000000000002},"last":3.1869999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38700000000000001,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67400000000000004,"5min":2.3079999999999998,"15min":6.1820000000000004},"last":3.2269999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.65100000000000002,"15min":0.69899999999999995},"min":{"1min":0.21099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.3469999999999995,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.255},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.58099999999999996},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.2489999999999997,"5min":8.1080000000000005,"15min":14.503},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4947984,"kb_used_data":3645192,"kb_used_omap":1018,"kb_used_meta":1301765,"kb_avail":89423856,"statfs":{"total":96636764160,"available":91570028544,"internally_reserved":0,"allocated":3732676608,"data_stored":8723288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1333007660},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,4,3,16,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":2.673,"15min":1.153},"min":{"1min":0.24199999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":1.899,"5min":207.047,"15min":207.047},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":2.7170000000000001,"15min":1.216},"min":{"1min":0.24299999999999999,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":2.6480000000000001,"5min":207.00200000000001,"15min":207.00200000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:41:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.53800000000000003,"15min":0.48599999999999999},"min":{"1min":0.27200000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.0999999999999996,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.0019999999999998,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3730833408,"data_stored":8722808832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3607154688,"data_stored":8458727443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7342112768,"data_stored":17185660947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:42:01.593 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:01.593 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:01.593 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:01.594 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:01.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:42:21.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:42:22.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:42:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:42:22.260 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2216,"stamp":"2023-10-04T01:42:21.619410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27191460409,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2015326,"num_read_kb":7273485,"num_write":1856406,"num_write_kb":7259163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96119,"ondisk_log_size":96119,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18832280,"kb_used_data":14408192,"kb_used_omap":4115,"kb_used_meta":4419948,"kb_avail":264283240,"statfs":{"total":289910292480,"available":270626037760,"internally_reserved":0,"allocated":14753988608,"data_stored":34436716154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4526027321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,7,12,7,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22831104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5698,"num_read_kb":22748,"num_write":5578,"num_write_kb":22312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005418"},"pg_stats":[{"pgid":"2.7","version":"143'7748","reported_seq":10975,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:19.335525+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:42:19.335525+0000","last_peered":"2023-10-04T01:42:19.335525+0000","last_clean":"2023-10-04T01:42:19.335525+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:42:19.335525+0000","last_undegraded":"2023-10-04T01:42:19.335525+0000","last_fullsized":"2023-10-04T01:42:19.335525+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7748,"log_dups_size":0,"ondisk_log_size":7748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'7123","reported_seq":10733,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:17.630440+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:42:17.630440+0000","last_peered":"2023-10-04T01:42:17.630440+0000","last_clean":"2023-10-04T01:42:17.630440+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:42:17.630440+0000","last_undegraded":"2023-10-04T01:42:17.630440+0000","last_fullsized":"2023-10-04T01:42:17.630440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'7889","reported_seq":11423,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:19.339891+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:42:19.339891+0000","last_peered":"2023-10-04T01:42:19.339891+0000","last_clean":"2023-10-04T01:42:19.339891+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:42:19.339891+0000","last_undegraded":"2023-10-04T01:42:19.339891+0000","last_fullsized":"2023-10-04T01:42:19.339891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7889,"log_dups_size":0,"ondisk_log_size":7889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'6320","reported_seq":10700,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:19.335426+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:42:19.335426+0000","last_peered":"2023-10-04T01:42:19.335426+0000","last_clean":"2023-10-04T01:42:19.335426+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:42:19.335426+0000","last_undegraded":"2023-10-04T01:42:19.335426+0000","last_fullsized":"2023-10-04T01:42:19.335426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6320,"log_dups_size":0,"ondisk_log_size":6320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"143'460651","reported_seq":908415,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:21.537215+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:42:21.537215+0000","last_peered":"2023-10-04T01:42:21.537215+0000","last_clean":"2023-10-04T01:42:21.537215+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:42:21.537215+0000","last_undegraded":"2023-10-04T01:42:21.537215+0000","last_fullsized":"2023-10-04T01:42:21.537215+0000","mapping_epoch":15,"log_start":"142'450573","ondisk_log_start":"142'450573","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1429,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 197s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6828900352,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":453060,"num_read_kb":1781876,"num_write":452868,"num_write_kb":1811472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'16329","reported_seq":21276,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:20.298907+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:42:20.298907+0000","last_peered":"2023-10-04T01:42:20.298907+0000","last_clean":"2023-10-04T01:42:20.298907+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:42:20.298907+0000","last_undegraded":"2023-10-04T01:42:20.298907+0000","last_fullsized":"2023-10-04T01:42:20.298907+0000","mapping_epoch":12,"log_start":"77'6300","ondisk_log_start":"77'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"143'450232","reported_seq":888535,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:19.771208+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:42:19.771208+0000","last_peered":"2023-10-04T01:42:19.771208+0000","last_clean":"2023-10-04T01:42:19.771208+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:42:19.771208+0000","last_undegraded":"2023-10-04T01:42:19.771208+0000","last_fullsized":"2023-10-04T01:42:19.771208+0000","mapping_epoch":15,"log_start":"142'440162","ondisk_log_start":"142'440162","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1518,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 192s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6684778496,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":443496,"num_read_kb":1744448,"num_write":442503,"num_write_kb":1770012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8745,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:15.194048+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:42:15.194048+0000","last_peered":"2023-10-04T01:42:15.194048+0000","last_clean":"2023-10-04T01:42:15.194048+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:42:15.194048+0000","last_undegraded":"2023-10-04T01:42:15.194048+0000","last_fullsized":"2023-10-04T01:42:15.194048+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"143'470515","reported_seq":929170,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:21.523491+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:42:21.523491+0000","last_peered":"2023-10-04T01:42:21.523491+0000","last_clean":"2023-10-04T01:42:21.523491+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:42:21.523491+0000","last_undegraded":"2023-10-04T01:42:21.523491+0000","last_fullsized":"2023-10-04T01:42:21.523491+0000","mapping_epoch":15,"log_start":"142'460464","ondisk_log_start":"142'460464","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1329,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 193s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":6995357696,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":464096,"num_read_kb":1826140,"num_write":462670,"num_write_kb":1850680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9357,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:15.179851+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:42:15.179851+0000","last_peered":"2023-10-04T01:42:15.179851+0000","last_clean":"2023-10-04T01:42:15.179851+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:42:15.179851+0000","last_undegraded":"2023-10-04T01:42:15.179851+0000","last_fullsized":"2023-10-04T01:42:15.179851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"143'449233","reported_seq":887473,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:19.774132+0000","last_change":"2023-10-04T01:42:16.114805+0000","last_active":"2023-10-04T01:42:19.774132+0000","last_peered":"2023-10-04T01:42:19.774132+0000","last_clean":"2023-10-04T01:42:19.774132+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:42:19.774132+0000","last_undegraded":"2023-10-04T01:42:19.774132+0000","last_fullsized":"2023-10-04T01:42:19.774132+0000","mapping_epoch":15,"log_start":"142'439163","ondisk_log_start":"142'439163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6681817107,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":443500,"num_read_kb":1744168,"num_write":442721,"num_write_kb":1770878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"143'17211","reported_seq":22380,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:42:19.358872+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:42:19.358872+0000","last_peered":"2023-10-04T01:42:19.358872+0000","last_clean":"2023-10-04T01:42:19.358872+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:42:19.358872+0000","last_undegraded":"2023-10-04T01:42:19.358872+0000","last_fullsized":"2023-10-04T01:42:19.358872+0000","mapping_epoch":12,"log_start":"77'7200","ondisk_log_start":"77'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":315,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:18.156432+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:18.156432+0000","last_peered":"2023-10-04T01:41:18.156432+0000","last_clean":"2023-10-04T01:41:18.156432+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:18.156432+0000","last_undegraded":"2023-10-04T01:41:18.156432+0000","last_fullsized":"2023-10-04T01:41:18.156432+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27190853651,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1804152,"num_read_kb":7096632,"num_write":1800762,"num_write_kb":7203042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14748762112,"data_stored":34435858470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40269,"ondisk_log_size":40269,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55751,"ondisk_log_size":55751,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8985112,"kb_used_data":7200848,"kb_used_omap":2099,"kb_used_meta":1782156,"kb_avail":85386728,"statfs":{"total":96636764160,"available":87436009472,"internally_reserved":0,"allocated":7373668352,"data_stored":17215977473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1824928626},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.41099999999999998,"15min":0.38300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.2779999999999996,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.443},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":5.319,"5min":5.319,"15min":5.319},"last":0.41599999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.51500000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.657,"5min":1.657,"15min":6.7709999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.45200000000000001,"15min":0.46200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.9509999999999996,"5min":7.415,"15min":7.415},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739198,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4838684,"kb_used_data":3543316,"kb_used_omap":997,"kb_used_meta":1294362,"kb_avail":89533156,"statfs":{"total":96636764160,"available":91681951744,"internally_reserved":0,"allocated":3628355584,"data_stored":8478162321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1325427099},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.36699999999999999,"15min":0.373},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":5.7240000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38700000000000001,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67400000000000004,"5min":2.3079999999999998,"15min":6.1820000000000004},"last":0.44900000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:41:29 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.65100000000000002,"15min":0.69899999999999995},"min":{"1min":0.21099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.3469999999999995,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52400000000000002,"15min":0.58099999999999996},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.2489999999999997,"5min":8.1080000000000005,"15min":14.503},"last":0.27700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5008484,"kb_used_data":3664028,"kb_used_omap":1018,"kb_used_meta":1343429,"kb_avail":89363356,"statfs":{"total":96636764160,"available":91508076544,"internally_reserved":0,"allocated":3751964672,"data_stored":8742576360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1375671596},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,9,7,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":1.1479999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.52800000000000002,"5min":1.899,"15min":207.047},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41599999999999998,"15min":1.2170000000000001},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":2.6480000000000001,"15min":207.00200000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:41:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.53800000000000003,"15min":0.48599999999999999},"min":{"1min":0.27200000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.0999999999999996,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.379},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.0019999999999998,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3750043648,"data_stored":8742019072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3625885696,"data_stored":8477458451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7372832768,"data_stored":17216380947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:42:22.262 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:22.262 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:22.262 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:22.262 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:22.262 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:42:42.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:42:42.821 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:42:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:42:42.867 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2227,"stamp":"2023-10-04T01:42:41.622885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27217621561,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2021851,"num_read_kb":7299445,"num_write":1862760,"num_write_kb":7284579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96230,"ondisk_log_size":96230,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19082348,"kb_used_data":14462924,"kb_used_omap":4115,"kb_used_meta":4615276,"kb_avail":264033172,"statfs":{"total":289910292480,"available":270369968128,"internally_reserved":0,"allocated":14810034176,"data_stored":34492738682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4726043193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,6,12,30,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18644992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4679,"num_read_kb":18620,"num_write":4533,"num_write_kb":18132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002052"},"pg_stats":[{"pgid":"2.7","version":"144'7776","reported_seq":11012,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.832378+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:42:39.832378+0000","last_peered":"2023-10-04T01:42:39.832378+0000","last_clean":"2023-10-04T01:42:39.832378+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:42:39.832378+0000","last_undegraded":"2023-10-04T01:42:39.832378+0000","last_fullsized":"2023-10-04T01:42:39.832378+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'7137","reported_seq":10752,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.641869+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:42:39.641869+0000","last_peered":"2023-10-04T01:42:39.641869+0000","last_clean":"2023-10-04T01:42:39.641869+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:42:39.641869+0000","last_undegraded":"2023-10-04T01:42:39.641869+0000","last_fullsized":"2023-10-04T01:42:39.641869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'7917","reported_seq":11461,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.832609+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:42:39.832609+0000","last_peered":"2023-10-04T01:42:39.832609+0000","last_clean":"2023-10-04T01:42:39.832609+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:42:39.832609+0000","last_undegraded":"2023-10-04T01:42:39.832609+0000","last_fullsized":"2023-10-04T01:42:39.832609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'6348","reported_seq":10738,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.825968+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:42:39.825968+0000","last_peered":"2023-10-04T01:42:39.825968+0000","last_clean":"2023-10-04T01:42:39.825968+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:42:39.825968+0000","last_undegraded":"2023-10-04T01:42:39.825968+0000","last_fullsized":"2023-10-04T01:42:39.825968+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'462131","reported_seq":911284,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:36.574799+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:42:36.574799+0000","last_peered":"2023-10-04T01:42:36.574799+0000","last_clean":"2023-10-04T01:42:36.574799+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:42:36.574799+0000","last_undegraded":"2023-10-04T01:42:36.574799+0000","last_fullsized":"2023-10-04T01:42:36.574799+0000","mapping_epoch":15,"log_start":"142'452073","ondisk_log_start":"142'452073","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1520,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 212s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6834692096,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":454442,"num_read_kb":1787392,"num_write":454276,"num_write_kb":1817104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'16372","reported_seq":21330,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:35.659014+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:42:35.659014+0000","last_peered":"2023-10-04T01:42:35.659014+0000","last_clean":"2023-10-04T01:42:35.659014+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:42:35.659014+0000","last_undegraded":"2023-10-04T01:42:35.659014+0000","last_fullsized":"2023-10-04T01:42:35.659014+0000","mapping_epoch":12,"log_start":"77'6300","ondisk_log_start":"77'6300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'452179","reported_seq":892394,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:39.840938+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:42:39.840938+0000","last_peered":"2023-10-04T01:42:39.840938+0000","last_clean":"2023-10-04T01:42:39.840938+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:42:39.840938+0000","last_undegraded":"2023-10-04T01:42:39.840938+0000","last_fullsized":"2023-10-04T01:42:39.840938+0000","mapping_epoch":15,"log_start":"142'442162","ondisk_log_start":"142'442162","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1657,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 212s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6692347904,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":445394,"num_read_kb":1752000,"num_write":444335,"num_write_kb":1777340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8762,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:35.712830+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:42:35.712830+0000","last_peered":"2023-10-04T01:42:35.712830+0000","last_clean":"2023-10-04T01:42:35.712830+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:42:35.712830+0000","last_undegraded":"2023-10-04T01:42:35.712830+0000","last_fullsized":"2023-10-04T01:42:35.712830+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'471955","reported_seq":931986,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:36.546626+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:42:36.546626+0000","last_peered":"2023-10-04T01:42:36.546626+0000","last_clean":"2023-10-04T01:42:36.546626+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:42:36.546626+0000","last_undegraded":"2023-10-04T01:42:36.546626+0000","last_fullsized":"2023-10-04T01:42:36.546626+0000","mapping_epoch":15,"log_start":"142'461864","ondisk_log_start":"142'461864","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1438,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 208s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7000903680,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":465463,"num_read_kb":1831568,"num_write":464023,"num_write_kb":1856092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9370,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:35.669932+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:42:35.669932+0000","last_peered":"2023-10-04T01:42:35.669932+0000","last_clean":"2023-10-04T01:42:35.669932+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:42:35.669932+0000","last_undegraded":"2023-10-04T01:42:35.669932+0000","last_fullsized":"2023-10-04T01:42:35.669932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'450994","reported_seq":891119,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.842167+0000","last_change":"2023-10-04T01:42:37.957157+0000","last_active":"2023-10-04T01:42:39.842167+0000","last_peered":"2023-10-04T01:42:39.842167+0000","last_clean":"2023-10-04T01:42:39.842167+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:42:39.842167+0000","last_undegraded":"2023-10-04T01:42:39.842167+0000","last_fullsized":"2023-10-04T01:42:39.842167+0000","mapping_epoch":15,"log_start":"142'440963","ondisk_log_start":"142'440963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6689071123,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":445378,"num_read_kb":1751632,"num_write":444482,"num_write_kb":1777922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'17253","reported_seq":22434,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:39.705785+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:42:39.705785+0000","last_peered":"2023-10-04T01:42:39.705785+0000","last_clean":"2023-10-04T01:42:39.705785+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:42:39.705785+0000","last_undegraded":"2023-10-04T01:42:39.705785+0000","last_fullsized":"2023-10-04T01:42:39.705785+0000","mapping_epoch":12,"log_start":"77'7200","ondisk_log_start":"77'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":316,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.538635+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:39.538635+0000","last_peered":"2023-10-04T01:41:39.538635+0000","last_clean":"2023-10-04T01:41:39.538635+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:39.538635+0000","last_undegraded":"2023-10-04T01:41:39.538635+0000","last_fullsized":"2023-10-04T01:41:39.538635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27217014803,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1810677,"num_read_kb":7122592,"num_write":1807116,"num_write_kb":7228458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14804647936,"data_stored":34491744294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":55934,"ondisk_log_size":55934,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739203,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9100568,"kb_used_data":7230220,"kb_used_omap":2099,"kb_used_meta":1868236,"kb_avail":85271272,"statfs":{"total":96636764160,"available":87317782528,"internally_reserved":0,"allocated":7403745280,"data_stored":17246046721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1913074546},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:41:34 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.41099999999999998,"15min":0.38300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.2779999999999996,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.443},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":5.319,"5min":5.319,"15min":5.319},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.375,"15min":0.496},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56200000000000006,"5min":1.657,"15min":6.7709999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.438,"15min":0.45900000000000002},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":7.415},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359739202,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4913608,"kb_used_data":3557564,"kb_used_omap":997,"kb_used_meta":1355034,"kb_avail":89458232,"statfs":{"total":96636764160,"available":91605229568,"internally_reserved":0,"allocated":3642945536,"data_stored":8492744593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1387555227},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,0,3,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.41199999999999998,"15min":0.499},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.57799999999999996,"15min":0.70199999999999996},"min":{"1min":0.22500000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.0379999999999998,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46500000000000002,"15min":0.57899999999999996},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.6509999999999998,"5min":6.3520000000000003,"15min":14.503},"last":0.36099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5068172,"kb_used_data":3675140,"kb_used_omap":1018,"kb_used_meta":1392005,"kb_avail":89303668,"statfs":{"total":96636764160,"available":91446956032,"internally_reserved":0,"allocated":3763343360,"data_stored":8753947368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1425413420},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":1.1479999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.52800000000000002,"5min":1.899,"15min":207.047},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41599999999999998,"15min":1.2170000000000001},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":2.6480000000000001,"15min":207.00200000000001},"last":0.255}]},{"osd":2,"last update":"Wed Oct 4 01:42:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.54600000000000004,"15min":0.47999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51800000000000002,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.317},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.0710000000000002,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3761364992,"data_stored":8753340416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3640602624,"data_stored":8492175379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7402680320,"data_stored":17246228499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:42:42.869 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:42.869 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:42.869 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:42.869 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:42:42.870 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:42:56.588 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:42:56.579+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:42:56.588 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:42:56.579+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:42:56.588 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:42:56.579+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:43:02.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:43:03.448 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:43:03.449 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:43:03.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2237,"stamp":"2023-10-04T01:43:01.626407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27248067129,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2029274,"num_read_kb":7329021,"num_write":1870122,"num_write_kb":7314027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96358,"ondisk_log_size":96358,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19302324,"kb_used_data":14521948,"kb_used_omap":4115,"kb_used_meta":4776236,"kb_avail":263813196,"statfs":{"total":289910292480,"available":270144712704,"internally_reserved":0,"allocated":14870474752,"data_stored":34553179258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4890866233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,14,6,9,7,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18694144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4576,"num_read_kb":18232,"num_write":4518,"num_write_kb":18072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002154"},"pg_stats":[{"pgid":"2.7","version":"144'7804","reported_seq":11048,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:58.947558+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:42:58.947558+0000","last_peered":"2023-10-04T01:42:58.947558+0000","last_clean":"2023-10-04T01:42:58.947558+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:42:58.947558+0000","last_undegraded":"2023-10-04T01:42:58.947558+0000","last_fullsized":"2023-10-04T01:42:58.947558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7804,"log_dups_size":0,"ondisk_log_size":7804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'7152","reported_seq":10772,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:57.995811+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:42:57.995811+0000","last_peered":"2023-10-04T01:42:57.995811+0000","last_clean":"2023-10-04T01:42:57.995811+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:42:57.995811+0000","last_undegraded":"2023-10-04T01:42:57.995811+0000","last_fullsized":"2023-10-04T01:42:57.995811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7152,"log_dups_size":0,"ondisk_log_size":7152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'7946","reported_seq":11498,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:58.938635+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:42:58.938635+0000","last_peered":"2023-10-04T01:42:58.938635+0000","last_clean":"2023-10-04T01:42:58.938635+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:42:58.938635+0000","last_undegraded":"2023-10-04T01:42:58.938635+0000","last_fullsized":"2023-10-04T01:42:58.938635+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7946,"log_dups_size":0,"ondisk_log_size":7946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'6377","reported_seq":10775,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:58.944787+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:42:58.944787+0000","last_peered":"2023-10-04T01:42:58.944787+0000","last_clean":"2023-10-04T01:42:58.944787+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:42:58.944787+0000","last_undegraded":"2023-10-04T01:42:58.944787+0000","last_fullsized":"2023-10-04T01:42:58.944787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'464164","reported_seq":915194,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:56.718494+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:42:56.718494+0000","last_peered":"2023-10-04T01:42:56.718494+0000","last_clean":"2023-10-04T01:42:56.718494+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:42:56.718494+0000","last_undegraded":"2023-10-04T01:42:56.718494+0000","last_fullsized":"2023-10-04T01:42:56.718494+0000","mapping_epoch":15,"log_start":"142'454073","ondisk_log_start":"142'454073","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1679,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 232s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6842572800,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":456307,"num_read_kb":1794832,"num_write":456180,"num_write_kb":1824720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'16415","reported_seq":21385,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:56.138929+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:42:56.138929+0000","last_peered":"2023-10-04T01:42:56.138929+0000","last_clean":"2023-10-04T01:42:56.138929+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:42:56.138929+0000","last_undegraded":"2023-10-04T01:42:56.138929+0000","last_fullsized":"2023-10-04T01:42:56.138929+0000","mapping_epoch":12,"log_start":"77'6400","ondisk_log_start":"77'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'454087","reported_seq":896186,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:59.966918+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:42:59.966918+0000","last_peered":"2023-10-04T01:42:59.966918+0000","last_clean":"2023-10-04T01:42:59.966918+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:42:59.966918+0000","last_undegraded":"2023-10-04T01:42:59.966918+0000","last_fullsized":"2023-10-04T01:42:59.966918+0000","mapping_epoch":15,"log_start":"142'444063","ondisk_log_start":"142'444063","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1793,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 232s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6699806720,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":447264,"num_read_kb":1759440,"num_write":446133,"num_write_kb":1784532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8778,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:56.139096+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:42:56.139096+0000","last_peered":"2023-10-04T01:42:56.139096+0000","last_clean":"2023-10-04T01:42:56.139096+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:42:56.139096+0000","last_undegraded":"2023-10-04T01:42:56.139096+0000","last_fullsized":"2023-10-04T01:42:56.139096+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'473916","reported_seq":935880,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:42:56.716025+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:42:56.716025+0000","last_peered":"2023-10-04T01:42:56.716025+0000","last_clean":"2023-10-04T01:42:56.716025+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:42:56.716025+0000","last_undegraded":"2023-10-04T01:42:56.716025+0000","last_fullsized":"2023-10-04T01:42:56.716025+0000","mapping_epoch":15,"log_start":"142'463864","ondisk_log_start":"142'463864","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1611,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7008432128,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":467383,"num_read_kb":1839204,"num_write":465843,"num_write_kb":1863372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9382,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:56.159453+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:42:56.159453+0000","last_peered":"2023-10-04T01:42:56.159453+0000","last_clean":"2023-10-04T01:42:56.159453+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:42:56.159453+0000","last_undegraded":"2023-10-04T01:42:56.159453+0000","last_fullsized":"2023-10-04T01:42:56.159453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'452834","reported_seq":894738,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:59.965049+0000","last_change":"2023-10-04T01:42:57.101340+0000","last_active":"2023-10-04T01:42:59.965049+0000","last_peered":"2023-10-04T01:42:59.965049+0000","last_clean":"2023-10-04T01:42:59.965049+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:42:59.965049+0000","last_undegraded":"2023-10-04T01:42:59.965049+0000","last_fullsized":"2023-10-04T01:42:59.965049+0000","mapping_epoch":15,"log_start":"142'442763","ondisk_log_start":"142'442763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6696648723,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":447146,"num_read_kb":1758692,"num_write":446322,"num_write_kb":1785282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'17296","reported_seq":22491,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:58.982856+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:42:58.982856+0000","last_peered":"2023-10-04T01:42:58.982856+0000","last_clean":"2023-10-04T01:42:58.982856+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:42:58.982856+0000","last_undegraded":"2023-10-04T01:42:58.982856+0000","last_fullsized":"2023-10-04T01:42:58.982856+0000","mapping_epoch":12,"log_start":"77'7200","ondisk_log_start":"77'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":316,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.538635+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:39.538635+0000","last_peered":"2023-10-04T01:41:39.538635+0000","last_clean":"2023-10-04T01:41:39.538635+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:39.538635+0000","last_undegraded":"2023-10-04T01:41:39.538635+0000","last_fullsized":"2023-10-04T01:41:39.538635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27247460371,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1818100,"num_read_kb":7152168,"num_write":1814478,"num_write_kb":7257906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14864887808,"data_stored":34551984166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40238,"ondisk_log_size":40238,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56021,"ondisk_log_size":56021,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739207,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9219020,"kb_used_data":7259844,"kb_used_omap":2099,"kb_used_meta":1957068,"kb_avail":85152820,"statfs":{"total":96636764160,"available":87196487680,"internally_reserved":0,"allocated":7434080256,"data_stored":17276381697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":2004038514},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.437,"15min":0.39100000000000001},"min":{"1min":0.219,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.4279999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.44900000000000001},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.22900000000000001},"max":{"1min":1.46,"5min":5.319,"15min":5.319},"last":0.41799999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.375,"15min":0.496},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56200000000000006,"5min":1.657,"15min":6.7709999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.438,"15min":0.45900000000000002},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":7.415},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739206,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4964220,"kb_used_data":3572084,"kb_used_omap":997,"kb_used_meta":1391130,"kb_avail":89407620,"statfs":{"total":96636764160,"available":91553402880,"internally_reserved":0,"allocated":3657814016,"data_stored":8507613073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1424517531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,6,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.63},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.41199999999999998,"15min":0.499},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.57799999999999996,"15min":0.70199999999999996},"min":{"1min":0.22500000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.0379999999999998,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46500000000000002,"15min":0.57899999999999996},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.6509999999999998,"5min":6.3520000000000003,"15min":14.503},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5119084,"kb_used_data":3690020,"kb_used_omap":1018,"kb_used_meta":1428037,"kb_avail":89252756,"statfs":{"total":96636764160,"available":91394822144,"internally_reserved":0,"allocated":3778580480,"data_stored":8769184488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1462310188},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,2,4,6,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":1.1479999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.52800000000000002,"5min":1.899,"15min":207.047},"last":0.372},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41599999999999998,"15min":1.2170000000000001},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":2.6480000000000001,"15min":207.00200000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:42:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.54600000000000004,"15min":0.47999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51800000000000002,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.0710000000000002,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3776618496,"data_stored":8768593920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3655503872,"data_stored":8507076627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7432765440,"data_stored":17276313619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:43:03.498 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:03.498 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:03.499 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:03.499 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:03.499 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:43:16.422 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:16.407+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:43:16.423 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:16.415+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:43:21.111 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.103+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:43:21.117 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.107+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:43:21.117 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.107+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:43:21.120 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.115+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:43:21.137 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.131+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:43:21.137 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.131+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:43:21.137 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.131+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:43:21.137 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:43:21.131+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:43:23.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:43:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:43:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:43:24.117 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2248,"stamp":"2023-10-04T01:43:23.630246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27282444857,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2037652,"num_read_kb":7362337,"num_write":1878459,"num_write_kb":7347375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96398,"ondisk_log_size":96398,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19575596,"kb_used_data":14585300,"kb_used_omap":4115,"kb_used_meta":4986156,"kb_avail":263539924,"statfs":{"total":289910292480,"available":269864882176,"internally_reserved":0,"allocated":14935347200,"data_stored":34618051706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":5105824313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,6,7,9,11,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18698240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4609,"num_read_kb":18344,"num_write":4541,"num_write_kb":18164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002098"},"pg_stats":[{"pgid":"2.7","version":"144'7822","reported_seq":11075,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:19.076834+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:43:19.076834+0000","last_peered":"2023-10-04T01:43:19.076834+0000","last_clean":"2023-10-04T01:43:19.076834+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:43:19.076834+0000","last_undegraded":"2023-10-04T01:43:19.076834+0000","last_fullsized":"2023-10-04T01:43:19.076834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7822,"log_dups_size":0,"ondisk_log_size":7822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'7162","reported_seq":10786,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:17.214768+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:43:17.214768+0000","last_peered":"2023-10-04T01:43:17.214768+0000","last_clean":"2023-10-04T01:43:17.214768+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:43:17.214768+0000","last_undegraded":"2023-10-04T01:43:17.214768+0000","last_fullsized":"2023-10-04T01:43:17.214768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7162,"log_dups_size":0,"ondisk_log_size":7162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'7965","reported_seq":11525,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:19.076862+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:43:19.076862+0000","last_peered":"2023-10-04T01:43:19.076862+0000","last_clean":"2023-10-04T01:43:19.076862+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:43:19.076862+0000","last_undegraded":"2023-10-04T01:43:19.076862+0000","last_fullsized":"2023-10-04T01:43:19.076862+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":7965,"log_dups_size":0,"ondisk_log_size":7965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'6396","reported_seq":10802,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:19.077058+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:43:19.077058+0000","last_peered":"2023-10-04T01:43:19.077058+0000","last_clean":"2023-10-04T01:43:19.077058+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:43:19.077058+0000","last_undegraded":"2023-10-04T01:43:19.077058+0000","last_fullsized":"2023-10-04T01:43:19.077058+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'466635","reported_seq":920042,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:21.814568+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:43:21.814568+0000","last_peered":"2023-10-04T01:43:21.814568+0000","last_clean":"2023-10-04T01:43:21.814568+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:43:21.814568+0000","last_undegraded":"2023-10-04T01:43:21.814568+0000","last_fullsized":"2023-10-04T01:43:21.814568+0000","mapping_epoch":15,"log_start":"143'456573","ondisk_log_start":"143'456573","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1880,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 257s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6852153344,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":458669,"num_read_kb":1804240,"num_write":458488,"num_write_kb":1833952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'16457","reported_seq":21442,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:21.738767+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:43:21.738767+0000","last_peered":"2023-10-04T01:43:21.738767+0000","last_clean":"2023-10-04T01:43:21.738767+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:43:21.738767+0000","last_undegraded":"2023-10-04T01:43:21.738767+0000","last_fullsized":"2023-10-04T01:43:21.738767+0000","mapping_epoch":12,"log_start":"77'6400","ondisk_log_start":"77'6400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'456031","reported_seq":899992,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:20.029144+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:43:20.029144+0000","last_peered":"2023-10-04T01:43:20.029144+0000","last_clean":"2023-10-04T01:43:20.029144+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:43:20.029144+0000","last_undegraded":"2023-10-04T01:43:20.029144+0000","last_fullsized":"2023-10-04T01:43:20.029144+0000","mapping_epoch":15,"log_start":"143'445963","ondisk_log_start":"143'445963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1913,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 252s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6707367936,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":449118,"num_read_kb":1766800,"num_write":447979,"num_write_kb":1791916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8794,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:16.626778+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:43:16.626778+0000","last_peered":"2023-10-04T01:43:16.626778+0000","last_clean":"2023-10-04T01:43:16.626778+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:43:16.626778+0000","last_undegraded":"2023-10-04T01:43:16.626778+0000","last_fullsized":"2023-10-04T01:43:16.626778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'476415","reported_seq":940785,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:21.806238+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:43:21.806238+0000","last_peered":"2023-10-04T01:43:21.806238+0000","last_clean":"2023-10-04T01:43:21.806238+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:43:21.806238+0000","last_undegraded":"2023-10-04T01:43:21.806238+0000","last_fullsized":"2023-10-04T01:43:21.806238+0000","mapping_epoch":15,"log_start":"143'466364","ondisk_log_start":"143'466364","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1741,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7018283008,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":469777,"num_read_kb":1848740,"num_write":468236,"num_write_kb":1872944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9394,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:16.622786+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:43:16.622786+0000","last_peered":"2023-10-04T01:43:16.622786+0000","last_clean":"2023-10-04T01:43:16.622786+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:43:16.622786+0000","last_undegraded":"2023-10-04T01:43:16.622786+0000","last_fullsized":"2023-10-04T01:43:16.622786+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'454624","reported_seq":898303,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:20.028345+0000","last_change":"2023-10-04T01:43:19.295887+0000","last_active":"2023-10-04T01:43:20.028345+0000","last_peered":"2023-10-04T01:43:20.028345+0000","last_clean":"2023-10-04T01:43:20.028345+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:43:20.028345+0000","last_undegraded":"2023-10-04T01:43:20.028345+0000","last_fullsized":"2023-10-04T01:43:20.028345+0000","mapping_epoch":15,"log_start":"143'444563","ondisk_log_start":"143'444563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6704033811,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":448914,"num_read_kb":1765704,"num_write":448112,"num_write_kb":1792442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'17324","reported_seq":22531,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:19.094095+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:43:19.094095+0000","last_peered":"2023-10-04T01:43:19.094095+0000","last_clean":"2023-10-04T01:43:19.094095+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:43:19.094095+0000","last_undegraded":"2023-10-04T01:43:19.094095+0000","last_fullsized":"2023-10-04T01:43:19.094095+0000","mapping_epoch":12,"log_start":"77'7300","ondisk_log_start":"77'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":316,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-04T01:41:39.538635+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:41:39.538635+0000","last_peered":"2023-10-04T01:41:39.538635+0000","last_clean":"2023-10-04T01:41:39.538635+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:41:39.538635+0000","last_undegraded":"2023-10-04T01:41:39.538635+0000","last_fullsized":"2023-10-04T01:41:39.538635+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27281838099,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1826478,"num_read_kb":7185484,"num_write":1822815,"num_write_kb":7291254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14929608704,"data_stored":34616705062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40242,"ondisk_log_size":40242,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56057,"ondisk_log_size":56057,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739211,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9344472,"kb_used_data":7289232,"kb_used_omap":2099,"kb_used_meta":2053132,"kb_avail":85027368,"statfs":{"total":96636764160,"available":87068024832,"internally_reserved":0,"allocated":7464173568,"data_stored":17306475009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":2102408050},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.437,"15min":0.39100000000000001},"min":{"1min":0.219,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.4279999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.311},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.44900000000000001},"min":{"1min":0.27400000000000002,"5min":0.24099999999999999,"15min":0.22900000000000001},"max":{"1min":1.46,"5min":5.319,"15min":5.319},"last":0.34899999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.375,"15min":0.496},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56200000000000006,"5min":1.657,"15min":6.7709999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.438,"15min":0.45900000000000002},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":7.415},"last":0.45700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739210,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5032980,"kb_used_data":3586700,"kb_used_omap":997,"kb_used_meta":1445274,"kb_avail":89338860,"statfs":{"total":96636764160,"available":91482992640,"internally_reserved":0,"allocated":3672780800,"data_stored":8522579857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1479960987},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.189,"5min":0.187,"15min":0.17899999999999999},"max":{"1min":3.1869999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.41199999999999998,"15min":0.499},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.46100000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:42:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.57799999999999996,"15min":0.70199999999999996},"min":{"1min":0.22500000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.0379999999999998,"5min":8.3469999999999995,"15min":14.467000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46500000000000002,"15min":0.57899999999999996},"min":{"1min":0.26500000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.6509999999999998,"5min":6.3520000000000003,"15min":14.503},"last":0.42699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739211,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5198144,"kb_used_data":3709368,"kb_used_omap":1018,"kb_used_meta":1487749,"kb_avail":89173696,"statfs":{"total":96636764160,"available":91313864704,"internally_reserved":0,"allocated":3798392832,"data_stored":8788996840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1523455276},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,1,4,6,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":1.1479999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.52800000000000002,"5min":1.899,"15min":207.047},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.41599999999999998,"15min":1.2170000000000001},"min":{"1min":0.23000000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":2.6480000000000001,"15min":207.00200000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:42:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.54600000000000004,"15min":0.47999999999999998},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.51800000000000002,"5min":5.2329999999999997,"15min":5.2329999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.0710000000000002,"5min":5.4710000000000001,"15min":7.3300000000000001},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3795869696,"data_stored":8787845120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3670384640,"data_stored":8521957395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7463354368,"data_stored":17306902547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:43:24.119 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:24.119 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:24.119 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:24.119 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:24.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:43:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:43:24.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:43:25.419 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:43:25.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:43:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:43:26.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:43:26.646 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:43:26.680 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:43:46.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:43:47.246 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:43:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:43:47.296 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2259,"stamp":"2023-10-04T01:43:45.639930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27315544633,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2045802,"num_read_kb":7394837,"num_write":1886496,"num_write_kb":7379523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96456,"ondisk_log_size":96456,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19318312,"kb_used_data":14652944,"kb_used_omap":4115,"kb_used_meta":4661228,"kb_avail":263797208,"statfs":{"total":289910292480,"available":270128340992,"internally_reserved":0,"allocated":15004614656,"data_stored":34687319162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4773098041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,8,4,9,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18771968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4665,"num_read_kb":18580,"num_write":4561,"num_write_kb":18244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008115"},"pg_stats":[{"pgid":"2.7","version":"144'7862","reported_seq":11125,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:44.801398+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:43:44.801398+0000","last_peered":"2023-10-04T01:43:44.801398+0000","last_clean":"2023-10-04T01:43:44.801398+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:43:44.801398+0000","last_undegraded":"2023-10-04T01:43:44.801398+0000","last_fullsized":"2023-10-04T01:43:44.801398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'7182","reported_seq":10811,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:43.172331+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:43:43.172331+0000","last_peered":"2023-10-04T01:43:43.172331+0000","last_clean":"2023-10-04T01:43:43.172331+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:43:43.172331+0000","last_undegraded":"2023-10-04T01:43:43.172331+0000","last_fullsized":"2023-10-04T01:43:43.172331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7182,"log_dups_size":0,"ondisk_log_size":7182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'8005","reported_seq":11575,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:44.801553+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:43:44.801553+0000","last_peered":"2023-10-04T01:43:44.801553+0000","last_clean":"2023-10-04T01:43:44.801553+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:43:44.801553+0000","last_undegraded":"2023-10-04T01:43:44.801553+0000","last_fullsized":"2023-10-04T01:43:44.801553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'6436","reported_seq":10852,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:44.805007+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:43:44.805007+0000","last_peered":"2023-10-04T01:43:44.805007+0000","last_clean":"2023-10-04T01:43:44.805007+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:43:44.805007+0000","last_undegraded":"2023-10-04T01:43:44.805007+0000","last_fullsized":"2023-10-04T01:43:44.805007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"144'468579","reported_seq":923792,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:41.884031+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:43:41.884031+0000","last_peered":"2023-10-04T01:43:41.884031+0000","last_clean":"2023-10-04T01:43:41.884031+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:43:41.884031+0000","last_undegraded":"2023-10-04T01:43:41.884031+0000","last_fullsized":"2023-10-04T01:43:41.884031+0000","mapping_epoch":15,"log_start":"143'458573","ondisk_log_start":"143'458573","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1993,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 277s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6859784192,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":460462,"num_read_kb":1811384,"num_write":460340,"num_write_kb":1841360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"144'16511","reported_seq":21505,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:37.381549+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:43:37.381549+0000","last_peered":"2023-10-04T01:43:37.381549+0000","last_clean":"2023-10-04T01:43:37.381549+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:43:37.381549+0000","last_undegraded":"2023-10-04T01:43:37.381549+0000","last_fullsized":"2023-10-04T01:43:37.381549+0000","mapping_epoch":12,"log_start":"77'6500","ondisk_log_start":"77'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"144'458389","reported_seq":904625,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:45.223117+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:43:45.223117+0000","last_peered":"2023-10-04T01:43:45.223117+0000","last_clean":"2023-10-04T01:43:45.223117+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:43:45.223117+0000","last_undegraded":"2023-10-04T01:43:45.223117+0000","last_fullsized":"2023-10-04T01:43:45.223117+0000","mapping_epoch":15,"log_start":"143'448365","ondisk_log_start":"143'448365","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2091,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 277s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6716510208,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":451376,"num_read_kb":1775800,"num_write":450192,"num_write_kb":1800768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8814,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:42.224476+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:43:42.224476+0000","last_peered":"2023-10-04T01:43:42.224476+0000","last_clean":"2023-10-04T01:43:42.224476+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:43:42.224476+0000","last_undegraded":"2023-10-04T01:43:42.224476+0000","last_fullsized":"2023-10-04T01:43:42.224476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"144'478350","reported_seq":944602,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:43:41.865213+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:43:41.865213+0000","last_peered":"2023-10-04T01:43:41.865213+0000","last_clean":"2023-10-04T01:43:41.865213+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:43:41.865213+0000","last_undegraded":"2023-10-04T01:43:41.865213+0000","last_fullsized":"2023-10-04T01:43:41.865213+0000","mapping_epoch":15,"log_start":"143'468264","ondisk_log_start":"143'468264","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1903,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 273s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7025704960,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":471650,"num_read_kb":1856208,"num_write":470038,"num_write_kb":1880152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9410,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:42.220936+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:43:42.220936+0000","last_peered":"2023-10-04T01:43:42.220936+0000","last_clean":"2023-10-04T01:43:42.220936+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:43:42.220936+0000","last_undegraded":"2023-10-04T01:43:42.220936+0000","last_fullsized":"2023-10-04T01:43:42.220936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"144'456794","reported_seq":902705,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:45.219628+0000","last_change":"2023-10-04T01:43:37.464093+0000","last_active":"2023-10-04T01:43:45.219628+0000","last_peered":"2023-10-04T01:43:45.219628+0000","last_clean":"2023-10-04T01:43:45.219628+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:43:45.219628+0000","last_undegraded":"2023-10-04T01:43:45.219628+0000","last_fullsized":"2023-10-04T01:43:45.219628+0000","mapping_epoch":15,"log_start":"143'446763","ondisk_log_start":"143'446763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6712938515,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":451140,"num_read_kb":1774592,"num_write":450282,"num_write_kb":1801122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"144'17383","reported_seq":22605,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:44.872260+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:43:44.872260+0000","last_peered":"2023-10-04T01:43:44.872260+0000","last_clean":"2023-10-04T01:43:44.872260+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:43:44.872260+0000","last_undegraded":"2023-10-04T01:43:44.872260+0000","last_fullsized":"2023-10-04T01:43:44.872260+0000","mapping_epoch":12,"log_start":"77'7300","ondisk_log_start":"77'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":317,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:42:28.182297+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:42:28.182297+0000","last_peered":"2023-10-04T01:42:28.182297+0000","last_clean":"2023-10-04T01:42:28.182297+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:42:28.182297+0000","last_undegraded":"2023-10-04T01:42:28.182297+0000","last_fullsized":"2023-10-04T01:42:28.182297+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27314937875,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1834628,"num_read_kb":7217984,"num_write":1830852,"num_write_kb":7323402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14999011328,"data_stored":34686107686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56210,"ondisk_log_size":56210,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8928968,"kb_used_data":7324864,"kb_used_omap":2099,"kb_used_meta":1601996,"kb_avail":85442872,"statfs":{"total":96636764160,"available":87493500928,"internally_reserved":0,"allocated":7500660736,"data_stored":17342962177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1640444786},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.437,"15min":0.38800000000000001},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.64400000000000002,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.497},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.49199999999999999,"15min":0.44600000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.67400000000000004,"5min":5.319,"15min":5.319},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38300000000000001,"15min":0.49099999999999999},"min":{"1min":0.26600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.72899999999999998,"5min":1.657,"15min":6.7709999999999999},"last":0.252},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47999999999999998,"15min":0.45500000000000002},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.0579999999999998,"5min":6.0890000000000004,"15min":7.415},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739215,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5122532,"kb_used_data":3604124,"kb_used_omap":997,"kb_used_meta":1517402,"kb_avail":89249308,"statfs":{"total":96636764160,"available":91391291392,"internally_reserved":0,"allocated":3690622976,"data_stored":8540422033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1553820059},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.63,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.49299999999999999},"min":{"1min":0.23100000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.052,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":0.81499999999999995,"15min":0.79500000000000004},"min":{"1min":0.27500000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":10.494999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.28,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.29899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.531000000000001,"5min":10.531000000000001,"15min":14.503},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5266812,"kb_used_data":3723956,"kb_used_omap":1018,"kb_used_meta":1541829,"kb_avail":89105028,"statfs":{"total":96636764160,"available":91243548672,"internally_reserved":0,"allocated":3813330944,"data_stored":8803934952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1578833196},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:43:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.438,"15min":1.161},"min":{"1min":0.21099999999999999,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":5.7969999999999997,"5min":5.7969999999999997,"15min":207.047},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.497,"15min":1.23},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":6.069,"5min":6.069,"15min":207.00200000000001},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:43:28 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.60099999999999998,"15min":0.502},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.433,"15min":0.49099999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3810836480,"data_stored":8802811904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3688337408,"data_stored":8539910163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7499837440,"data_stored":17343385619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:43:47.298 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:47.298 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:47.298 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:47.298 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:43:47.299 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:44:07.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:44:07.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:44:07.885 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:44:07.917 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2271,"stamp":"2023-10-04T01:44:07.643791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27349582393,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2054340,"num_read_kb":7428901,"num_write":1894806,"num_write_kb":7412763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96590,"ondisk_log_size":96590,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19550740,"kb_used_data":14716220,"kb_used_omap":4115,"kb_used_meta":4830380,"kb_avail":263564780,"statfs":{"total":289910292480,"available":269890334720,"internally_reserved":0,"allocated":15069409280,"data_stored":34752090617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":4946309689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,10,9,6,18,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19091456,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4750,"num_read_kb":18956,"num_write":4654,"num_write_kb":18616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002158"},"pg_stats":[{"pgid":"2.7","version":"145'7883","reported_seq":11155,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:03.646891+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:44:03.646891+0000","last_peered":"2023-10-04T01:44:03.646891+0000","last_clean":"2023-10-04T01:44:03.646891+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:44:03.646891+0000","last_undegraded":"2023-10-04T01:44:03.646891+0000","last_fullsized":"2023-10-04T01:44:03.646891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'7193","reported_seq":10827,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:03.613504+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:44:03.613504+0000","last_peered":"2023-10-04T01:44:03.613504+0000","last_clean":"2023-10-04T01:44:03.613504+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:44:03.613504+0000","last_undegraded":"2023-10-04T01:44:03.613504+0000","last_fullsized":"2023-10-04T01:44:03.613504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7193,"log_dups_size":0,"ondisk_log_size":7193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'8026","reported_seq":11605,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:03.650341+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:44:03.650341+0000","last_peered":"2023-10-04T01:44:03.650341+0000","last_clean":"2023-10-04T01:44:03.650341+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:44:03.650341+0000","last_undegraded":"2023-10-04T01:44:03.650341+0000","last_fullsized":"2023-10-04T01:44:03.650341+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8026,"log_dups_size":0,"ondisk_log_size":8026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'6457","reported_seq":10882,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:03.645209+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:44:03.645209+0000","last_peered":"2023-10-04T01:44:03.645209+0000","last_clean":"2023-10-04T01:44:03.645209+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:44:03.645209+0000","last_undegraded":"2023-10-04T01:44:03.645209+0000","last_fullsized":"2023-10-04T01:44:03.645209+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'471026","reported_seq":928698,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:07.053339+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:44:07.053339+0000","last_peered":"2023-10-04T01:44:07.053339+0000","last_clean":"2023-10-04T01:44:07.053339+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:44:07.053339+0000","last_undegraded":"2023-10-04T01:44:07.053339+0000","last_fullsized":"2023-10-04T01:44:07.053339+0000","mapping_epoch":15,"log_start":"143'460973","ondisk_log_start":"143'460973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2174,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 302s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6869184512,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":462905,"num_read_kb":1821132,"num_write":462639,"num_write_kb":1850556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'16550","reported_seq":21560,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:02.703105+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:44:02.703105+0000","last_peered":"2023-10-04T01:44:02.703105+0000","last_clean":"2023-10-04T01:44:02.703105+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:44:02.703105+0000","last_undegraded":"2023-10-04T01:44:02.703105+0000","last_fullsized":"2023-10-04T01:44:02.703105+0000","mapping_epoch":12,"log_start":"77'6500","ondisk_log_start":"77'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'460318","reported_seq":908420,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:05.290151+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:44:05.290151+0000","last_peered":"2023-10-04T01:44:05.290151+0000","last_clean":"2023-10-04T01:44:05.290151+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:44:05.290151+0000","last_undegraded":"2023-10-04T01:44:05.290151+0000","last_fullsized":"2023-10-04T01:44:05.290151+0000","mapping_epoch":15,"log_start":"143'450265","ondisk_log_start":"143'450265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2232,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 297s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6723977216,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":453230,"num_read_kb":1783188,"num_write":452008,"num_write_kb":1808032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8831,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:02.708149+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:44:02.708149+0000","last_peered":"2023-10-04T01:44:02.708149+0000","last_clean":"2023-10-04T01:44:02.708149+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:44:02.708149+0000","last_undegraded":"2023-10-04T01:44:02.708149+0000","last_fullsized":"2023-10-04T01:44:02.708149+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'480855","reported_seq":949479,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:07.043449+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:44:07.043449+0000","last_peered":"2023-10-04T01:44:07.043449+0000","last_clean":"2023-10-04T01:44:07.043449+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:44:07.043449+0000","last_undegraded":"2023-10-04T01:44:07.043449+0000","last_fullsized":"2023-10-04T01:44:07.043449+0000","mapping_epoch":15,"log_start":"143'470764","ondisk_log_start":"143'470764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2046,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 298s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7035494400,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":474005,"num_read_kb":1865608,"num_write":472424,"num_write_kb":1889696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9423,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:02.720198+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:44:02.720198+0000","last_peered":"2023-10-04T01:44:02.720198+0000","last_clean":"2023-10-04T01:44:02.720198+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:44:02.720198+0000","last_undegraded":"2023-10-04T01:44:02.720198+0000","last_fullsized":"2023-10-04T01:44:02.720198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'458603","reported_seq":906406,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:05.288638+0000","last_change":"2023-10-04T01:44:05.287775+0000","last_active":"2023-10-04T01:44:05.288638+0000","last_peered":"2023-10-04T01:44:05.288638+0000","last_clean":"2023-10-04T01:44:05.288638+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:44:05.288638+0000","last_undegraded":"2023-10-04T01:44:05.288638+0000","last_fullsized":"2023-10-04T01:44:05.288638+0000","mapping_epoch":15,"log_start":"143'448563","ondisk_log_start":"143'448563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6720319507,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":453026,"num_read_kb":1782120,"num_write":452091,"num_write_kb":1808358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'17414","reported_seq":22649,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:03.695547+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:44:03.695547+0000","last_peered":"2023-10-04T01:44:03.695547+0000","last_clean":"2023-10-04T01:44:03.695547+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:44:03.695547+0000","last_undegraded":"2023-10-04T01:44:03.695547+0000","last_fullsized":"2023-10-04T01:44:03.695547+0000","mapping_epoch":12,"log_start":"77'7400","ondisk_log_start":"77'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":318,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:40.189002+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:43:40.189002+0000","last_peered":"2023-10-04T01:43:40.189002+0000","last_clean":"2023-10-04T01:43:40.189002+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:43:40.189002+0000","last_undegraded":"2023-10-04T01:43:40.189002+0000","last_fullsized":"2023-10-04T01:43:40.189002+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27348975635,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1843166,"num_read_kb":7252048,"num_write":1839162,"num_write_kb":7356642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15063093248,"data_stored":34750189606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40237,"ondisk_log_size":40237,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56254,"ondisk_log_size":56254,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739220,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9069188,"kb_used_data":7354236,"kb_used_omap":2099,"kb_used_meta":1712844,"kb_avail":85302652,"statfs":{"total":96636764160,"available":87349915648,"internally_reserved":0,"allocated":7530737664,"data_stored":17373031382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1753953138},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.437,"15min":0.38800000000000001},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.64400000000000002,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.49199999999999999,"15min":0.44600000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.67400000000000004,"5min":5.319,"15min":5.319},"last":0.38600000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38300000000000001,"15min":0.49099999999999999},"min":{"1min":0.26600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.72899999999999998,"5min":1.657,"15min":6.7709999999999999},"last":0.308},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47999999999999998,"15min":0.45500000000000002},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.0579999999999998,"5min":6.0890000000000004,"15min":7.415},"last":1.4350000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739219,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5155744,"kb_used_data":3619288,"kb_used_omap":997,"kb_used_meta":1535450,"kb_avail":89216096,"statfs":{"total":96636764160,"available":91357282304,"internally_reserved":0,"allocated":3706150912,"data_stored":8555942246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1572301211},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,0,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.63,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.49299999999999999},"min":{"1min":0.23100000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.052,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.39900000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":0.81499999999999995,"15min":0.79500000000000004},"min":{"1min":0.27500000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":10.494999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.246},{"interface":"front","average":{"1min":1.28,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.29899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.531000000000001,"5min":10.531000000000001,"15min":14.503},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5325808,"kb_used_data":3742696,"kb_used_omap":1018,"kb_used_meta":1582085,"kb_avail":89046032,"statfs":{"total":96636764160,"available":91183136768,"internally_reserved":0,"allocated":3832520704,"data_stored":8823116989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1620055340},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,7,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:43:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.438,"15min":1.161},"min":{"1min":0.21099999999999999,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":5.7969999999999997,"5min":5.7969999999999997,"15min":207.047},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.497,"15min":1.23},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":6.069,"5min":6.069,"15min":207.00200000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Wed Oct 4 01:43:28 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.60099999999999998,"15min":0.502},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.433,"15min":0.49099999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3830022144,"data_stored":8821997568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3703189504,"data_stored":8554762259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7529881600,"data_stored":17373429779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:44:07.920 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:07.921 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:07.921 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:07.921 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:07.922 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:44:27.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:44:28.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:44:28.653 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:44:28.693 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2281,"stamp":"2023-10-04T01:44:27.647315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27379548729,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2061614,"num_read_kb":7457881,"num_write":1902106,"num_write_kb":7441963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96732,"ondisk_log_size":96732,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19874732,"kb_used_data":14773780,"kb_used_omap":4115,"kb_used_meta":5096812,"kb_avail":263240788,"statfs":{"total":289910292480,"available":269558566912,"internally_reserved":0,"allocated":15128350720,"data_stored":34811032057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":5219136057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,11,10,16,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19001344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4605,"num_read_kb":18364,"num_write":4631,"num_write_kb":18524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002138"},"pg_stats":[{"pgid":"2.7","version":"145'7905","reported_seq":11185,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.182765+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:44:23.182765+0000","last_peered":"2023-10-04T01:44:23.182765+0000","last_clean":"2023-10-04T01:44:23.182765+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:44:23.182765+0000","last_undegraded":"2023-10-04T01:44:23.182765+0000","last_fullsized":"2023-10-04T01:44:23.182765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7905,"log_dups_size":0,"ondisk_log_size":7905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'7204","reported_seq":10842,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:22.593502+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:44:22.593502+0000","last_peered":"2023-10-04T01:44:22.593502+0000","last_clean":"2023-10-04T01:44:22.593502+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:44:22.593502+0000","last_undegraded":"2023-10-04T01:44:22.593502+0000","last_fullsized":"2023-10-04T01:44:22.593502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'8048","reported_seq":11635,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.182778+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:44:23.182778+0000","last_peered":"2023-10-04T01:44:23.182778+0000","last_clean":"2023-10-04T01:44:23.182778+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:44:23.182778+0000","last_undegraded":"2023-10-04T01:44:23.182778+0000","last_fullsized":"2023-10-04T01:44:23.182778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8048,"log_dups_size":0,"ondisk_log_size":8048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'6479","reported_seq":10912,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.186423+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:44:23.186423+0000","last_peered":"2023-10-04T01:44:23.186423+0000","last_clean":"2023-10-04T01:44:23.186423+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:44:23.186423+0000","last_undegraded":"2023-10-04T01:44:23.186423+0000","last_fullsized":"2023-10-04T01:44:23.186423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'472983","reported_seq":932461,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:27.136539+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:44:27.136539+0000","last_peered":"2023-10-04T01:44:27.136539+0000","last_clean":"2023-10-04T01:44:27.136539+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:44:27.136539+0000","last_undegraded":"2023-10-04T01:44:27.136539+0000","last_fullsized":"2023-10-04T01:44:27.136539+0000","mapping_epoch":15,"log_start":"144'462973","ondisk_log_start":"144'462973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2361,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 322s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6876573696,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":464703,"num_read_kb":1828304,"num_write":464443,"num_write_kb":1857772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'16589","reported_seq":21611,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:25.769027+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:44:25.769027+0000","last_peered":"2023-10-04T01:44:25.769027+0000","last_clean":"2023-10-04T01:44:25.769027+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:44:25.769027+0000","last_undegraded":"2023-10-04T01:44:25.769027+0000","last_fullsized":"2023-10-04T01:44:25.769027+0000","mapping_epoch":12,"log_start":"77'6500","ondisk_log_start":"77'6500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'462232","reported_seq":912184,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:25.431436+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:44:25.431436+0000","last_peered":"2023-10-04T01:44:25.431436+0000","last_clean":"2023-10-04T01:44:25.431436+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:44:25.431436+0000","last_undegraded":"2023-10-04T01:44:25.431436+0000","last_fullsized":"2023-10-04T01:44:25.431436+0000","mapping_epoch":15,"log_start":"144'452165","ondisk_log_start":"144'452165","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2378,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6731354112,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":455068,"num_read_kb":1790508,"num_write":453803,"num_write_kb":1815212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8848,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.189350+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:44:23.189350+0000","last_peered":"2023-10-04T01:44:23.189350+0000","last_clean":"2023-10-04T01:44:23.189350+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:44:23.189350+0000","last_undegraded":"2023-10-04T01:44:23.189350+0000","last_fullsized":"2023-10-04T01:44:23.189350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'482847","reported_seq":953348,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:27.129765+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:44:27.129765+0000","last_peered":"2023-10-04T01:44:27.129765+0000","last_clean":"2023-10-04T01:44:27.129765+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:44:27.129765+0000","last_undegraded":"2023-10-04T01:44:27.129765+0000","last_fullsized":"2023-10-04T01:44:27.129765+0000","mapping_epoch":15,"log_start":"144'472764","ondisk_log_start":"144'472764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2194,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 319s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7043186688,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":475873,"num_read_kb":1873056,"num_write":474295,"num_write_kb":1897180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9435,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.187793+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:44:23.187793+0000","last_peered":"2023-10-04T01:44:23.187793+0000","last_clean":"2023-10-04T01:44:23.187793+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:44:23.187793+0000","last_undegraded":"2023-10-04T01:44:23.187793+0000","last_fullsized":"2023-10-04T01:44:23.187793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'460433","reported_seq":910011,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:25.437948+0000","last_change":"2023-10-04T01:44:25.357975+0000","last_active":"2023-10-04T01:44:25.437948+0000","last_peered":"2023-10-04T01:44:25.437948+0000","last_clean":"2023-10-04T01:44:25.437948+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:44:25.437948+0000","last_undegraded":"2023-10-04T01:44:25.437948+0000","last_fullsized":"2023-10-04T01:44:25.437948+0000","mapping_epoch":15,"log_start":"144'450363","ondisk_log_start":"144'450363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6727827475,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":454796,"num_read_kb":1789160,"num_write":453921,"num_write_kb":1815678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'17447","reported_seq":22694,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:23.182826+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:44:23.182826+0000","last_peered":"2023-10-04T01:44:23.182826+0000","last_clean":"2023-10-04T01:44:23.182826+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:44:23.182826+0000","last_undegraded":"2023-10-04T01:44:23.182826+0000","last_fullsized":"2023-10-04T01:44:23.182826+0000","mapping_epoch":12,"log_start":"77'7400","ondisk_log_start":"77'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":318,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:40.189002+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:43:40.189002+0000","last_peered":"2023-10-04T01:43:40.189002+0000","last_clean":"2023-10-04T01:43:40.189002+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:43:40.189002+0000","last_undegraded":"2023-10-04T01:43:40.189002+0000","last_fullsized":"2023-10-04T01:43:40.189002+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27378941971,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1850440,"num_read_kb":7281028,"num_write":1846462,"num_write_kb":7385842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15122726912,"data_stored":34809823270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40230,"ondisk_log_size":40230,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56403,"ondisk_log_size":56403,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9202448,"kb_used_data":7383368,"kb_used_omap":2099,"kb_used_meta":1816972,"kb_avail":85169392,"statfs":{"total":96636764160,"available":87213457408,"internally_reserved":0,"allocated":7560568832,"data_stored":17402862550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1860580210},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.437,"15min":0.38800000000000001},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.64400000000000002,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.49199999999999999,"15min":0.44600000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.67400000000000004,"5min":5.319,"15min":5.319},"last":0.40100000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38300000000000001,"15min":0.49099999999999999},"min":{"1min":0.26600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.72899999999999998,"5min":1.657,"15min":6.7709999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.47999999999999998,"15min":0.45500000000000002},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.0579999999999998,"5min":6.0890000000000004,"15min":7.415},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739223,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5242340,"kb_used_data":3633756,"kb_used_omap":997,"kb_used_meta":1607578,"kb_avail":89129500,"statfs":{"total":96636764160,"available":91268608000,"internally_reserved":0,"allocated":3720966144,"data_stored":8570757478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1646160283},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.63,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41899999999999998,"15min":0.49299999999999999},"min":{"1min":0.23100000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":2.052,"5min":3.2269999999999999,"15min":6.1820000000000004},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:43:33 2023","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":0.81499999999999995,"15min":0.79500000000000004},"min":{"1min":0.27500000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":10.494999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.28,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.29899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":10.531000000000001,"5min":10.531000000000001,"15min":14.503},"last":0.67400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5429944,"kb_used_data":3756656,"kb_used_omap":1018,"kb_used_meta":1672261,"kb_avail":88941896,"statfs":{"total":96636764160,"available":91076501504,"internally_reserved":0,"allocated":3846815744,"data_stored":8837412029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1712395564},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,7,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.436,"15min":1.141},"min":{"1min":0.19700000000000001,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.81499999999999995,"5min":5.7969999999999997,"15min":207.047},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.51100000000000001,"15min":1.2290000000000001},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":1.8160000000000001,"5min":6.069,"15min":207.00200000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:43:28 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.60099999999999998,"15min":0.502},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.433,"15min":0.49099999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3845070848,"data_stored":8837046272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3718037504,"data_stored":8569610259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7559618560,"data_stored":17403166739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:44:28.695 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:28.696 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:28.696 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:28.696 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:28.696 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:44:48.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:44:49.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:44:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:44:49.449 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2291,"stamp":"2023-10-04T01:44:47.650601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27409834553,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2069130,"num_read_kb":7487781,"num_write":1909481,"num_write_kb":7471463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96676,"ondisk_log_size":96676,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20074136,"kb_used_data":14832960,"kb_used_omap":4115,"kb_used_meta":5237036,"kb_avail":263041384,"statfs":{"total":289910292480,"available":269354377216,"internally_reserved":0,"allocated":15188951040,"data_stored":34871632377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":5362725433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,5,12,17,18,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19005440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4736,"num_read_kb":18840,"num_write":4619,"num_write_kb":18476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001943"},"pg_stats":[{"pgid":"2.7","version":"145'7929","reported_seq":11217,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:45.140597+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:44:45.140597+0000","last_peered":"2023-10-04T01:44:45.140597+0000","last_clean":"2023-10-04T01:44:45.140597+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:44:45.140597+0000","last_undegraded":"2023-10-04T01:44:45.140597+0000","last_fullsized":"2023-10-04T01:44:45.140597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'7216","reported_seq":10858,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:43.659259+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:44:43.659259+0000","last_peered":"2023-10-04T01:44:43.659259+0000","last_clean":"2023-10-04T01:44:43.659259+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:44:43.659259+0000","last_undegraded":"2023-10-04T01:44:43.659259+0000","last_fullsized":"2023-10-04T01:44:43.659259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7216,"log_dups_size":0,"ondisk_log_size":7216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'8072","reported_seq":11668,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:45.142985+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:44:45.142985+0000","last_peered":"2023-10-04T01:44:45.142985+0000","last_clean":"2023-10-04T01:44:45.142985+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:44:45.142985+0000","last_undegraded":"2023-10-04T01:44:45.142985+0000","last_fullsized":"2023-10-04T01:44:45.142985+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'6503","reported_seq":10945,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:45.141970+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:44:45.141970+0000","last_peered":"2023-10-04T01:44:45.141970+0000","last_clean":"2023-10-04T01:44:45.141970+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:44:45.141970+0000","last_undegraded":"2023-10-04T01:44:45.141970+0000","last_fullsized":"2023-10-04T01:44:45.141970+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6503,"log_dups_size":0,"ondisk_log_size":6503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"145'474983","reported_seq":936388,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:47.235931+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:44:47.235931+0000","last_peered":"2023-10-04T01:44:47.235931+0000","last_clean":"2023-10-04T01:44:47.235931+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:44:47.235931+0000","last_undegraded":"2023-10-04T01:44:47.235931+0000","last_fullsized":"2023-10-04T01:44:47.235931+0000","mapping_epoch":15,"log_start":"144'464973","ondisk_log_start":"144'464973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2458,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 343s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6884474880,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":466617,"num_read_kb":1835916,"num_write":466366,"num_write_kb":1865464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"145'16613","reported_seq":21648,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:44.700099+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:44:44.700099+0000","last_peered":"2023-10-04T01:44:44.700099+0000","last_clean":"2023-10-04T01:44:44.700099+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:44:44.700099+0000","last_undegraded":"2023-10-04T01:44:44.700099+0000","last_fullsized":"2023-10-04T01:44:44.700099+0000","mapping_epoch":12,"log_start":"77'6600","ondisk_log_start":"77'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"145'464171","reported_seq":915923,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:45.506950+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:44:45.506950+0000","last_peered":"2023-10-04T01:44:45.506950+0000","last_clean":"2023-10-04T01:44:45.506950+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:44:45.506950+0000","last_undegraded":"2023-10-04T01:44:45.506950+0000","last_fullsized":"2023-10-04T01:44:45.506950+0000","mapping_epoch":15,"log_start":"144'454165","ondisk_log_start":"144'454165","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2550,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 338s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6738784256,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":456855,"num_read_kb":1797592,"num_write":455605,"num_write_kb":1822420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8864,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:44.035606+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:44:44.035606+0000","last_peered":"2023-10-04T01:44:44.035606+0000","last_clean":"2023-10-04T01:44:44.035606+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:44:44.035606+0000","last_undegraded":"2023-10-04T01:44:44.035606+0000","last_fullsized":"2023-10-04T01:44:44.035606+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"145'484824","reported_seq":957279,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:44:47.224532+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:44:47.224532+0000","last_peered":"2023-10-04T01:44:47.224532+0000","last_clean":"2023-10-04T01:44:47.224532+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:44:47.224532+0000","last_undegraded":"2023-10-04T01:44:47.224532+0000","last_fullsized":"2023-10-04T01:44:47.224532+0000","mapping_epoch":15,"log_start":"144'474764","ondisk_log_start":"144'474764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2332,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 339s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7050850304,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":477812,"num_read_kb":1880780,"num_write":476161,"num_write_kb":1904644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9447,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:43.662670+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:44:43.662670+0000","last_peered":"2023-10-04T01:44:43.662670+0000","last_clean":"2023-10-04T01:44:43.662670+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:44:43.662670+0000","last_undegraded":"2023-10-04T01:44:43.662670+0000","last_fullsized":"2023-10-04T01:44:43.662670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"145'462217","reported_seq":913678,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:45.497907+0000","last_change":"2023-10-04T01:44:45.394852+0000","last_active":"2023-10-04T01:44:45.497907+0000","last_peered":"2023-10-04T01:44:45.497907+0000","last_clean":"2023-10-04T01:44:45.497907+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:44:45.497907+0000","last_undegraded":"2023-10-04T01:44:45.497907+0000","last_fullsized":"2023-10-04T01:44:45.497907+0000","mapping_epoch":15,"log_start":"144'452163","ondisk_log_start":"144'452163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6735118355,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":456672,"num_read_kb":1796640,"num_write":455705,"num_write_kb":1822814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'17483","reported_seq":22743,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:44:45.166901+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:44:45.166901+0000","last_peered":"2023-10-04T01:44:45.166901+0000","last_clean":"2023-10-04T01:44:45.166901+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:44:45.166901+0000","last_undegraded":"2023-10-04T01:44:45.166901+0000","last_fullsized":"2023-10-04T01:44:45.166901+0000","mapping_epoch":12,"log_start":"77'7400","ondisk_log_start":"77'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":318,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-04T01:43:40.189002+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:43:40.189002+0000","last_peered":"2023-10-04T01:43:40.189002+0000","last_clean":"2023-10-04T01:43:40.189002+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:43:40.189002+0000","last_undegraded":"2023-10-04T01:43:40.189002+0000","last_fullsized":"2023-10-04T01:43:40.189002+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27409227795,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1857956,"num_read_kb":7310928,"num_write":1853837,"num_write_kb":7415342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15183138816,"data_stored":34870235174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40130,"ondisk_log_size":40130,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56447,"ondisk_log_size":56447,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336064,"kb_used_data":7412856,"kb_used_omap":2099,"kb_used_meta":1921100,"kb_avail":85035776,"statfs":{"total":96636764160,"available":87076634624,"internally_reserved":0,"allocated":7590764544,"data_stored":17433058262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":1967207282},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.39200000000000002},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.75800000000000001,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.80000000000000004,"5min":5.319,"15min":5.319},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Oct 4 01:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.374,"15min":0.47699999999999998},"min":{"1min":0.21199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.77400000000000002,"5min":1.657,"15min":6.7709999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.48999999999999999,"15min":0.44700000000000001},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.4350000000000001,"5min":6.0890000000000004,"15min":7.415},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739227,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5274680,"kb_used_data":3648048,"kb_used_omap":997,"kb_used_meta":1625626,"kb_avail":89097160,"statfs":{"total":96636764160,"available":91235491840,"internally_reserved":0,"allocated":3735601152,"data_stored":8585392486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1664641435},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,4,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.36599999999999999,"15min":0.38},"min":{"1min":0.22700000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46000000000000002,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.45600000000000002,"15min":0.496},"min":{"1min":0.24399999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.63800000000000001,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.59799999999999998,"15min":0.63700000000000001},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.67400000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359739228,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5463392,"kb_used_data":3772056,"kb_used_omap":1018,"kb_used_meta":1690309,"kb_avail":88908448,"statfs":{"total":96636764160,"available":91042250752,"internally_reserved":0,"allocated":3862585344,"data_stored":8853181629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1730876716},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,0,6,11,7,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.436,"15min":1.141},"min":{"1min":0.19700000000000001,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.81499999999999995,"5min":5.7969999999999997,"15min":207.047},"last":0.188},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.51100000000000001,"15min":1.2290000000000001},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":1.8160000000000001,"5min":6.069,"15min":207.00200000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63200000000000001,"15min":0.51300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":5.7549999999999999,"5min":5.835,"15min":5.835},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.40999999999999998,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.37}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3860606976,"data_stored":8852582400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3732721664,"data_stored":8584294419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7589810176,"data_stored":17433358355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:44:49.451 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:49.452 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:49.452 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:49.452 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:44:49.453 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:45:09.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:45:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:45:09.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:45:10.026 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2303,"stamp":"2023-10-04T01:45:09.654830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27440153145,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2076646,"num_read_kb":7517677,"num_write":1916872,"num_write_kb":7501027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96769,"ondisk_log_size":96769,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20409188,"kb_used_data":14899212,"kb_used_omap":4115,"kb_used_meta":5505836,"kb_avail":262706332,"statfs":{"total":289910292480,"available":269011283968,"internally_reserved":0,"allocated":15256793088,"data_stored":34939451127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":5637976633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,4,15,15,12,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15323136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3831,"num_read_kb":15248,"num_write":3739,"num_write_kb":14956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002305"},"pg_stats":[{"pgid":"2.7","version":"146'7953","reported_seq":11250,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.546613+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:45:05.546613+0000","last_peered":"2023-10-04T01:45:05.546613+0000","last_clean":"2023-10-04T01:45:05.546613+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:45:05.546613+0000","last_undegraded":"2023-10-04T01:45:05.546613+0000","last_fullsized":"2023-10-04T01:45:05.546613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7953,"log_dups_size":0,"ondisk_log_size":7953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'7228","reported_seq":10876,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.315268+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:45:05.315268+0000","last_peered":"2023-10-04T01:45:05.315268+0000","last_clean":"2023-10-04T01:45:05.315268+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:45:05.315268+0000","last_undegraded":"2023-10-04T01:45:05.315268+0000","last_fullsized":"2023-10-04T01:45:05.315268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'8096","reported_seq":11701,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.546260+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:45:05.546260+0000","last_peered":"2023-10-04T01:45:05.546260+0000","last_clean":"2023-10-04T01:45:05.546260+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:45:05.546260+0000","last_undegraded":"2023-10-04T01:45:05.546260+0000","last_fullsized":"2023-10-04T01:45:05.546260+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'6527","reported_seq":10978,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.542746+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:45:05.542746+0000","last_peered":"2023-10-04T01:45:05.542746+0000","last_clean":"2023-10-04T01:45:05.542746+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:45:05.542746+0000","last_undegraded":"2023-10-04T01:45:05.542746+0000","last_fullsized":"2023-10-04T01:45:05.542746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6527,"log_dups_size":0,"ondisk_log_size":6527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'476976","reported_seq":940324,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:07.438803+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:45:07.438803+0000","last_peered":"2023-10-04T01:45:07.438803+0000","last_clean":"2023-10-04T01:45:07.438803+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:45:07.438803+0000","last_undegraded":"2023-10-04T01:45:07.438803+0000","last_fullsized":"2023-10-04T01:45:07.438803+0000","mapping_epoch":15,"log_start":"144'466973","ondisk_log_start":"144'466973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2571,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 363s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6892228608,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":468547,"num_read_kb":1843592,"num_write":468263,"num_write_kb":1873052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'16658","reported_seq":21706,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:06.734494+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:45:06.734494+0000","last_peered":"2023-10-04T01:45:06.734494+0000","last_clean":"2023-10-04T01:45:06.734494+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:45:06.734494+0000","last_undegraded":"2023-10-04T01:45:06.734494+0000","last_fullsized":"2023-10-04T01:45:06.734494+0000","mapping_epoch":12,"log_start":"77'6600","ondisk_log_start":"77'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'466129","reported_seq":919713,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:05.627170+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:45:05.627170+0000","last_peered":"2023-10-04T01:45:05.627170+0000","last_clean":"2023-10-04T01:45:05.627170+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:45:05.627170+0000","last_undegraded":"2023-10-04T01:45:05.627170+0000","last_fullsized":"2023-10-04T01:45:05.627170+0000","mapping_epoch":15,"log_start":"144'456065","ondisk_log_start":"144'456065","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2724,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 358s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6746238976,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":458678,"num_read_kb":1804836,"num_write":457419,"num_write_kb":1829676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8881,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:04.149290+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:45:04.149290+0000","last_peered":"2023-10-04T01:45:04.149290+0000","last_clean":"2023-10-04T01:45:04.149290+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:45:04.149290+0000","last_undegraded":"2023-10-04T01:45:04.149290+0000","last_fullsized":"2023-10-04T01:45:04.149290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'486799","reported_seq":961263,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:07.429724+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:45:07.429724+0000","last_peered":"2023-10-04T01:45:07.429724+0000","last_clean":"2023-10-04T01:45:07.429724+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:45:07.429724+0000","last_undegraded":"2023-10-04T01:45:07.429724+0000","last_fullsized":"2023-10-04T01:45:07.429724+0000","mapping_epoch":15,"log_start":"144'476764","ondisk_log_start":"144'476764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2451,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 359s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7058571264,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":479814,"num_read_kb":1888740,"num_write":478039,"num_write_kb":1912156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9460,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:04.139831+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:45:04.139831+0000","last_peered":"2023-10-04T01:45:04.139831+0000","last_clean":"2023-10-04T01:45:04.139831+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:45:04.139831+0000","last_undegraded":"2023-10-04T01:45:04.139831+0000","last_fullsized":"2023-10-04T01:45:04.139831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'464019","reported_seq":917250,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.579039+0000","last_change":"2023-10-04T01:45:04.570540+0000","last_active":"2023-10-04T01:45:05.579039+0000","last_peered":"2023-10-04T01:45:05.579039+0000","last_clean":"2023-10-04T01:45:05.579039+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:45:05.579039+0000","last_undegraded":"2023-10-04T01:45:05.579039+0000","last_fullsized":"2023-10-04T01:45:05.579039+0000","mapping_epoch":15,"log_start":"144'453963","ondisk_log_start":"144'453963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6742507539,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":458433,"num_read_kb":1803656,"num_write":457507,"num_write_kb":1830022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'17519","reported_seq":22792,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:05.573775+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:45:05.573775+0000","last_peered":"2023-10-04T01:45:05.573775+0000","last_clean":"2023-10-04T01:45:05.573775+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:45:05.573775+0000","last_undegraded":"2023-10-04T01:45:05.573775+0000","last_fullsized":"2023-10-04T01:45:05.573775+0000","mapping_epoch":12,"log_start":"77'7500","ondisk_log_start":"77'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":319,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:43:48.195754+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:43:48.195754+0000","last_peered":"2023-10-04T01:43:48.195754+0000","last_clean":"2023-10-04T01:43:48.195754+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:43:48.195754+0000","last_undegraded":"2023-10-04T01:43:48.195754+0000","last_fullsized":"2023-10-04T01:43:48.195754+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27439546387,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1865472,"num_read_kb":7340824,"num_write":1861228,"num_write_kb":7444906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15251337216,"data_stored":34938433574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40158,"ondisk_log_size":40158,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56512,"ondisk_log_size":56512,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9477152,"kb_used_data":7449496,"kb_used_omap":2099,"kb_used_meta":2025548,"kb_avail":84894688,"statfs":{"total":96636764160,"available":86932160512,"internally_reserved":0,"allocated":7628283904,"data_stored":17470569856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":2074162034},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.39200000000000002},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.75800000000000001,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.80000000000000004,"5min":5.319,"15min":5.319},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.374,"15min":0.47699999999999998},"min":{"1min":0.21199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.77400000000000002,"5min":1.657,"15min":6.7709999999999999},"last":0.498},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.48999999999999999,"15min":0.44700000000000001},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.4350000000000001,"5min":6.0890000000000004,"15min":7.415},"last":0.29299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5367824,"kb_used_data":3662536,"kb_used_omap":997,"kb_used_meta":1704282,"kb_avail":89004016,"statfs":{"total":96636764160,"available":91140112384,"internally_reserved":0,"allocated":3750436864,"data_stored":8600220432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1745185179},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,3,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.36599999999999999,"15min":0.38},"min":{"1min":0.22700000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46000000000000002,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.379},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.45600000000000002,"15min":0.496},"min":{"1min":0.24399999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":0.64600000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.63800000000000001,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.255},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.59799999999999998,"15min":0.63700000000000001},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.67400000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5564212,"kb_used_data":3787180,"kb_used_omap":1018,"kb_used_meta":1776005,"kb_avail":88807628,"statfs":{"total":96636764160,"available":90939011072,"internally_reserved":0,"allocated":3878072320,"data_stored":8868660839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1818629420},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,8,6,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:44:24 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.436,"15min":1.141},"min":{"1min":0.19700000000000001,"5min":0.19,"15min":0.18099999999999999},"max":{"1min":0.81499999999999995,"5min":5.7969999999999997,"15min":207.047},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.51100000000000001,"15min":1.2290000000000001},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":1.8160000000000001,"5min":6.069,"15min":207.00200000000001},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63200000000000001,"15min":0.51300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":5.7549999999999999,"5min":5.835,"15min":5.835},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.318,"5min":0.40999999999999998,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3876057088,"data_stored":8868032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3747536896,"data_stored":8599109651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7627743232,"data_stored":17471291411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:45:10.028 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:10.028 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:10.028 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:10.028 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:10.029 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:45:30.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:45:30.506 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:45:30.507 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:45:30.564 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2313,"stamp":"2023-10-04T01:45:29.658248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27469824569,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2083947,"num_read_kb":7546725,"num_write":1924111,"num_write_kb":7529983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96897,"ondisk_log_size":96897,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20602820,"kb_used_data":14956844,"kb_used_omap":4115,"kb_used_meta":5641836,"kb_avail":262512700,"statfs":{"total":289910292480,"available":268813004800,"internally_reserved":0,"allocated":15315808256,"data_stored":34998466295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4214215,"internal_metadata":5777240633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,4,10,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14831616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3646,"num_read_kb":14500,"num_write":3617,"num_write_kb":14468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002054"},"pg_stats":[{"pgid":"2.7","version":"146'7971","reported_seq":11277,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.619172+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:45:24.619172+0000","last_peered":"2023-10-04T01:45:24.619172+0000","last_clean":"2023-10-04T01:45:24.619172+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:45:24.619172+0000","last_undegraded":"2023-10-04T01:45:24.619172+0000","last_fullsized":"2023-10-04T01:45:24.619172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'7237","reported_seq":10889,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:23.994368+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:45:23.994368+0000","last_peered":"2023-10-04T01:45:23.994368+0000","last_clean":"2023-10-04T01:45:23.994368+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:45:23.994368+0000","last_undegraded":"2023-10-04T01:45:23.994368+0000","last_fullsized":"2023-10-04T01:45:23.994368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'8114","reported_seq":11727,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.618783+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:45:24.618783+0000","last_peered":"2023-10-04T01:45:24.618783+0000","last_clean":"2023-10-04T01:45:24.618783+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:45:24.618783+0000","last_undegraded":"2023-10-04T01:45:24.618783+0000","last_fullsized":"2023-10-04T01:45:24.618783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8114,"log_dups_size":0,"ondisk_log_size":8114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'6545","reported_seq":11004,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.618804+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:45:24.618804+0000","last_peered":"2023-10-04T01:45:24.618804+0000","last_clean":"2023-10-04T01:45:24.618804+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:45:24.618804+0000","last_undegraded":"2023-10-04T01:45:24.618804+0000","last_fullsized":"2023-10-04T01:45:24.618804+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'478872","reported_seq":944038,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:27.571143+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:45:27.571143+0000","last_peered":"2023-10-04T01:45:27.571143+0000","last_clean":"2023-10-04T01:45:27.571143+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:45:27.571143+0000","last_undegraded":"2023-10-04T01:45:27.571143+0000","last_fullsized":"2023-10-04T01:45:27.571143+0000","mapping_epoch":15,"log_start":"144'468773","ondisk_log_start":"144'468773","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2734,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 383s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6899453952,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":470367,"num_read_kb":1850836,"num_write":470026,"num_write_kb":1880104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'16685","reported_seq":21745,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:27.374530+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:45:27.374530+0000","last_peered":"2023-10-04T01:45:27.374530+0000","last_clean":"2023-10-04T01:45:27.374530+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:45:27.374530+0000","last_undegraded":"2023-10-04T01:45:27.374530+0000","last_fullsized":"2023-10-04T01:45:27.374530+0000","mapping_epoch":12,"log_start":"77'6600","ondisk_log_start":"77'6600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'467996","reported_seq":923342,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:25.690623+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:45:25.690623+0000","last_peered":"2023-10-04T01:45:25.690623+0000","last_clean":"2023-10-04T01:45:25.690623+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:45:25.690623+0000","last_undegraded":"2023-10-04T01:45:25.690623+0000","last_fullsized":"2023-10-04T01:45:25.690623+0000","mapping_epoch":15,"log_start":"144'457965","ondisk_log_start":"144'457965","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2869,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 378s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6753431552,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":460428,"num_read_kb":1811804,"num_write":459169,"num_write_kb":1836676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8897,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.624871+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:45:24.624871+0000","last_peered":"2023-10-04T01:45:24.624871+0000","last_clean":"2023-10-04T01:45:24.624871+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:45:24.624871+0000","last_undegraded":"2023-10-04T01:45:24.624871+0000","last_fullsized":"2023-10-04T01:45:24.624871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'488776","reported_seq":965153,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:27.571710+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:45:27.571710+0000","last_peered":"2023-10-04T01:45:27.571710+0000","last_clean":"2023-10-04T01:45:27.571710+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:45:27.571710+0000","last_undegraded":"2023-10-04T01:45:27.571710+0000","last_fullsized":"2023-10-04T01:45:27.571710+0000","mapping_epoch":15,"log_start":"144'478764","ondisk_log_start":"144'478764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2602,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 379s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7066173440,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":481719,"num_read_kb":1896324,"num_write":479894,"num_write_kb":1919576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9472,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.628511+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:45:24.628511+0000","last_peered":"2023-10-04T01:45:24.628511+0000","last_clean":"2023-10-04T01:45:24.628511+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:45:24.628511+0000","last_undegraded":"2023-10-04T01:45:24.628511+0000","last_fullsized":"2023-10-04T01:45:24.628511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'465890","reported_seq":920949,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:25.703395+0000","last_change":"2023-10-04T01:45:23.860560+0000","last_active":"2023-10-04T01:45:25.703395+0000","last_peered":"2023-10-04T01:45:25.703395+0000","last_clean":"2023-10-04T01:45:25.703395+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:45:25.703395+0000","last_undegraded":"2023-10-04T01:45:25.703395+0000","last_fullsized":"2023-10-04T01:45:25.703395+0000","mapping_epoch":15,"log_start":"144'455863","ondisk_log_start":"144'455863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6750158867,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":460259,"num_read_kb":1810908,"num_write":459378,"num_write_kb":1837506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'17546","reported_seq":22831,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:24.619032+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:45:24.619032+0000","last_peered":"2023-10-04T01:45:24.619032+0000","last_clean":"2023-10-04T01:45:24.619032+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:45:24.619032+0000","last_undegraded":"2023-10-04T01:45:24.619032+0000","last_fullsized":"2023-10-04T01:45:24.619032+0000","mapping_epoch":12,"log_start":"77'7500","ondisk_log_start":"77'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":319,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-04T01:43:48.195754+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:43:48.195754+0000","last_peered":"2023-10-04T01:43:48.195754+0000","last_clean":"2023-10-04T01:43:48.195754+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:43:48.195754+0000","last_undegraded":"2023-10-04T01:43:48.195754+0000","last_fullsized":"2023-10-04T01:43:48.195754+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27469217811,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1872773,"num_read_kb":7369872,"num_write":1868467,"num_write_kb":7473862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15310651392,"data_stored":34997747750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56629,"ondisk_log_size":56629,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739237,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9592300,"kb_used_data":7478564,"kb_used_omap":2099,"kb_used_meta":2111628,"kb_avail":84779540,"statfs":{"total":96636764160,"available":86814248960,"internally_reserved":0,"allocated":7658049536,"data_stored":17500335488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2149518,"internal_metadata":2162307954},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41199999999999998,"15min":0.39200000000000002},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.75800000000000001,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47799999999999998,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":0.80000000000000004,"5min":5.319,"15min":5.319},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.374,"15min":0.47699999999999998},"min":{"1min":0.21199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.77400000000000002,"5min":1.657,"15min":6.7709999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.48999999999999999,"15min":0.44700000000000001},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.4350000000000001,"5min":6.0890000000000004,"15min":7.415},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5395612,"kb_used_data":3676436,"kb_used_omap":997,"kb_used_meta":1718170,"kb_avail":88976228,"statfs":{"total":96636764160,"available":91111657472,"internally_reserved":0,"allocated":3764670464,"data_stored":8614454032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1759406491},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.36599999999999999,"15min":0.38},"min":{"1min":0.22700000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46000000000000002,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.45600000000000002,"15min":0.496},"min":{"1min":0.24399999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":1.0589999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.63800000000000001,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.249},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.59799999999999998,"15min":0.63700000000000001},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.67400000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.42299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5614908,"kb_used_data":3801844,"kb_used_omap":1018,"kb_used_meta":1812037,"kb_avail":88756932,"statfs":{"total":96636764160,"available":90887098368,"internally_reserved":0,"allocated":3893088256,"data_stored":8883676775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1855526188},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":1.1579999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":4.0419999999999998,"5min":5.7969999999999997,"15min":207.047},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":1.2230000000000001},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.217},"max":{"1min":2.137,"5min":6.069,"15min":207.00200000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63200000000000001,"15min":0.51300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":5.7549999999999999,"5min":5.835,"15min":5.835},"last":0.503},{"interface":"front","average":{"1min":0.318,"5min":0.40999999999999998,"15min":0.48199999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66400000000000003,"5min":5.0019999999999998,"15min":7.3300000000000001},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3890876416,"data_stored":8882851840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3762364416,"data_stored":8613937171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7657410560,"data_stored":17500958739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:45:30.566 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:30.566 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:30.566 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:30.567 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:30.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:45:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:45:31.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:45:31.774 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:45:31.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:45:32.397 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:45:32.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:45:33.007 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:45:33.038 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:45:53.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:45:53.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:45:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:45:53.654 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2324,"stamp":"2023-10-04T01:45:51.664036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27504439865,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2092130,"num_read_kb":7579265,"num_write":1932542,"num_write_kb":7563707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96918,"ondisk_log_size":96918,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20348000,"kb_used_data":15020676,"kb_used_omap":4397,"kb_used_meta":5322898,"kb_avail":262767520,"statfs":{"total":289910292480,"available":269073940480,"internally_reserved":0,"allocated":15381172224,"data_stored":35063830263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4503283,"internal_metadata":5450647821},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,4,5,9,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19443712,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4517,"num_read_kb":17976,"num_write":4732,"num_write_kb":18928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004099"},"pg_stats":[{"pgid":"2.7","version":"146'8005","reported_seq":11321,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.489963+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:45:50.489963+0000","last_peered":"2023-10-04T01:45:50.489963+0000","last_clean":"2023-10-04T01:45:50.489963+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:45:50.489963+0000","last_undegraded":"2023-10-04T01:45:50.489963+0000","last_fullsized":"2023-10-04T01:45:50.489963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'7254","reported_seq":10911,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.218860+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:45:50.218860+0000","last_peered":"2023-10-04T01:45:50.218860+0000","last_clean":"2023-10-04T01:45:50.218860+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:45:50.218860+0000","last_undegraded":"2023-10-04T01:45:50.218860+0000","last_fullsized":"2023-10-04T01:45:50.218860+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7254,"log_dups_size":0,"ondisk_log_size":7254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'8148","reported_seq":11771,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.484619+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:45:50.484619+0000","last_peered":"2023-10-04T01:45:50.484619+0000","last_clean":"2023-10-04T01:45:50.484619+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:45:50.484619+0000","last_undegraded":"2023-10-04T01:45:50.484619+0000","last_fullsized":"2023-10-04T01:45:50.484619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8148,"log_dups_size":0,"ondisk_log_size":8148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'6579","reported_seq":11048,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.476611+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:45:50.476611+0000","last_peered":"2023-10-04T01:45:50.476611+0000","last_clean":"2023-10-04T01:45:50.476611+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:45:50.476611+0000","last_undegraded":"2023-10-04T01:45:50.476611+0000","last_fullsized":"2023-10-04T01:45:50.476611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"146'480882","reported_seq":947910,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:47.702733+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:45:47.702733+0000","last_peered":"2023-10-04T01:45:47.702733+0000","last_clean":"2023-10-04T01:45:47.702733+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:45:47.702733+0000","last_undegraded":"2023-10-04T01:45:47.702733+0000","last_fullsized":"2023-10-04T01:45:47.702733+0000","mapping_epoch":15,"log_start":"145'470873","ondisk_log_start":"145'470873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2844,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 403s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6907338752,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":472206,"num_read_kb":1858136,"num_write":471947,"num_write_kb":1887788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"146'16716","reported_seq":21788,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:47.575279+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:45:47.575279+0000","last_peered":"2023-10-04T01:45:47.575279+0000","last_clean":"2023-10-04T01:45:47.575279+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:45:47.575279+0000","last_undegraded":"2023-10-04T01:45:47.575279+0000","last_fullsized":"2023-10-04T01:45:47.575279+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"146'470401","reported_seq":927984,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:50.844596+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:45:50.844596+0000","last_peered":"2023-10-04T01:45:50.844596+0000","last_clean":"2023-10-04T01:45:50.844596+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:45:50.844596+0000","last_undegraded":"2023-10-04T01:45:50.844596+0000","last_fullsized":"2023-10-04T01:45:50.844596+0000","mapping_epoch":15,"log_start":"145'460365","ondisk_log_start":"145'460365","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3016,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 403s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6762823680,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":462651,"num_read_kb":1820640,"num_write":461456,"num_write_kb":1845824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8913,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:45.121231+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:45:45.121231+0000","last_peered":"2023-10-04T01:45:45.121231+0000","last_clean":"2023-10-04T01:45:45.121231+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:45:45.121231+0000","last_undegraded":"2023-10-04T01:45:45.121231+0000","last_fullsized":"2023-10-04T01:45:45.121231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"146'490777","reported_seq":969038,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:45:47.703025+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:45:47.703025+0000","last_peered":"2023-10-04T01:45:47.703025+0000","last_clean":"2023-10-04T01:45:47.703025+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:45:47.703025+0000","last_undegraded":"2023-10-04T01:45:47.703025+0000","last_fullsized":"2023-10-04T01:45:47.703025+0000","mapping_epoch":15,"log_start":"145'480764","ondisk_log_start":"145'480764","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2703,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 399s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7074058240,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":483588,"num_read_kb":1903764,"num_write":481813,"num_write_kb":1927252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9485,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:45.129428+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:45:45.129428+0000","last_peered":"2023-10-04T01:45:45.129428+0000","last_clean":"2023-10-04T01:45:45.129428+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:45:45.129428+0000","last_undegraded":"2023-10-04T01:45:45.129428+0000","last_fullsized":"2023-10-04T01:45:45.129428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"146'468194","reported_seq":925519,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.850303+0000","last_change":"2023-10-04T01:45:50.244017+0000","last_active":"2023-10-04T01:45:50.850303+0000","last_peered":"2023-10-04T01:45:50.850303+0000","last_clean":"2023-10-04T01:45:50.850303+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:45:50.850303+0000","last_undegraded":"2023-10-04T01:45:50.850303+0000","last_fullsized":"2023-10-04T01:45:50.850303+0000","mapping_epoch":15,"log_start":"145'458163","ondisk_log_start":"145'458163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6759612435,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":462511,"num_read_kb":1819872,"num_write":461682,"num_write_kb":1846722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"146'17597","reported_seq":22897,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:50.512303+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:45:50.512303+0000","last_peered":"2023-10-04T01:45:50.512303+0000","last_clean":"2023-10-04T01:45:50.512303+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:45:50.512303+0000","last_undegraded":"2023-10-04T01:45:50.512303+0000","last_fullsized":"2023-10-04T01:45:50.512303+0000","mapping_epoch":12,"log_start":"77'7500","ondisk_log_start":"77'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":320,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:44:53.243449+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:44:53.243449+0000","last_peered":"2023-10-04T01:44:53.243449+0000","last_clean":"2023-10-04T01:44:53.243449+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:44:53.243449+0000","last_undegraded":"2023-10-04T01:44:53.243449+0000","last_fullsized":"2023-10-04T01:44:53.243449+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27503833107,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1880956,"num_read_kb":7402412,"num_write":1876898,"num_write_kb":7507586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15375659008,"data_stored":35062755366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40089,"ondisk_log_size":40089,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56730,"ondisk_log_size":56730,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9177648,"kb_used_data":7508324,"kb_used_omap":2381,"kb_used_meta":1666930,"kb_avail":85194192,"statfs":{"total":96636764160,"available":87238852608,"internally_reserved":0,"allocated":7688523776,"data_stored":17530809728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":1706936902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.44800000000000001,"15min":0.39900000000000002},"min":{"1min":0.17899999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.3799999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.19},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51000000000000001,"15min":0.46000000000000002},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":2.419,"5min":5.319,"15min":5.319},"last":0.25900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.47999999999999998},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57199999999999995,"5min":1.657,"15min":6.7709999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.48799999999999999,"15min":0.42599999999999999},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.61299999999999999,"5min":6.0890000000000004,"15min":7.415},"last":0.52200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739240,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5478900,"kb_used_data":3695020,"kb_used_omap":997,"kb_used_meta":1782874,"kb_avail":88892940,"statfs":{"total":96636764160,"available":91026370560,"internally_reserved":0,"allocated":3783700480,"data_stored":8633484048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1825663387},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.97899999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.222},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47899999999999998,"15min":0.49399999999999999},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":1.0589999999999999,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.76700000000000002,"15min":0.72199999999999998},"min":{"1min":0.23799999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":1.363,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.60199999999999998,"15min":0.622},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.60399999999999998,"5min":10.531000000000001,"15min":14.503},"last":0.27500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5691452,"kb_used_data":3817332,"kb_used_omap":1018,"kb_used_meta":1873093,"kb_avail":88680388,"statfs":{"total":96636764160,"available":90808717312,"internally_reserved":0,"allocated":3908947968,"data_stored":8899536487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1918047532},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,3,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":1.1579999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":4.0419999999999998,"5min":5.7969999999999997,"15min":207.047},"last":0.216},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":1.2230000000000001},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.217},"max":{"1min":2.137,"5min":6.069,"15min":207.00200000000001},"last":0.25}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":0.84399999999999997,"15min":0.56699999999999995},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":14.118,"5min":14.118,"15min":14.118},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.244,"5min":0.59099999999999997,"15min":0.53700000000000003},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.051,"5min":14.051,"15min":14.051},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3906600960,"data_stored":8898576384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3781173248,"data_stored":8632746003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7687884800,"data_stored":17531432979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:45:53.656 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:53.656 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:53.656 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:53.657 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:45:53.657 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:45:58.239 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.227+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:45:58.239 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.231+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:45:58.241 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.235+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:45:58.241 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.235+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:45:58.241 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.235+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:45:58.248 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.239+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:45:58.248 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.239+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:45:58.248 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.239+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:45:58.248 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.239+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:45:58.248 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:45:58.239+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:46:13.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:46:14.226 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:46:14.227 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:46:14.263 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2336,"stamp":"2023-10-04T01:46:13.671865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27538547257,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2100268,"num_read_kb":7611629,"num_write":1940861,"num_write_kb":7596983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97109,"ondisk_log_size":97109,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20629368,"kb_used_data":15083872,"kb_used_omap":4397,"kb_used_meta":5541074,"kb_avail":262486152,"statfs":{"total":289910292480,"available":268785819648,"internally_reserved":0,"allocated":15445884928,"data_stored":35128519540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4503283,"internal_metadata":5674060045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,10,18,15,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15065088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3540,"num_read_kb":14072,"num_write":3678,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.424260"},"pg_stats":[{"pgid":"2.7","version":"147'8029","reported_seq":11354,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.698951+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:46:10.698951+0000","last_peered":"2023-10-04T01:46:10.698951+0000","last_clean":"2023-10-04T01:46:10.698951+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:46:10.698951+0000","last_undegraded":"2023-10-04T01:46:10.698951+0000","last_fullsized":"2023-10-04T01:46:10.698951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8029,"log_dups_size":0,"ondisk_log_size":8029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'7266","reported_seq":10928,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.403088+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:46:10.403088+0000","last_peered":"2023-10-04T01:46:10.403088+0000","last_clean":"2023-10-04T01:46:10.403088+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:46:10.403088+0000","last_undegraded":"2023-10-04T01:46:10.403088+0000","last_fullsized":"2023-10-04T01:46:10.403088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'8172","reported_seq":11804,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.698852+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:46:10.698852+0000","last_peered":"2023-10-04T01:46:10.698852+0000","last_clean":"2023-10-04T01:46:10.698852+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:46:10.698852+0000","last_undegraded":"2023-10-04T01:46:10.698852+0000","last_fullsized":"2023-10-04T01:46:10.698852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8172,"log_dups_size":0,"ondisk_log_size":8172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'6603","reported_seq":11081,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.698786+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:46:10.698786+0000","last_peered":"2023-10-04T01:46:10.698786+0000","last_clean":"2023-10-04T01:46:10.698786+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:46:10.698786+0000","last_undegraded":"2023-10-04T01:46:10.698786+0000","last_fullsized":"2023-10-04T01:46:10.698786+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6603,"log_dups_size":0,"ondisk_log_size":6603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'483361","reported_seq":952674,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:12.969073+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:46:12.969073+0000","last_peered":"2023-10-04T01:46:12.969073+0000","last_clean":"2023-10-04T01:46:12.969073+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:46:12.969073+0000","last_undegraded":"2023-10-04T01:46:12.969073+0000","last_fullsized":"2023-10-04T01:46:12.969073+0000","mapping_epoch":15,"log_start":"145'473273","ondisk_log_start":"145'473273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3053,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 428s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6916784128,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":474481,"num_read_kb":1867176,"num_write":474255,"num_write_kb":1897020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'16764","reported_seq":21852,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:12.802834+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:46:12.802834+0000","last_peered":"2023-10-04T01:46:12.802834+0000","last_clean":"2023-10-04T01:46:12.802834+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:46:12.802834+0000","last_undegraded":"2023-10-04T01:46:12.802834+0000","last_fullsized":"2023-10-04T01:46:12.802834+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'472342","reported_seq":931700,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:10.921890+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:46:10.921890+0000","last_peered":"2023-10-04T01:46:10.921890+0000","last_clean":"2023-10-04T01:46:10.921890+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:46:10.921890+0000","last_undegraded":"2023-10-04T01:46:10.921890+0000","last_fullsized":"2023-10-04T01:46:10.921890+0000","mapping_epoch":15,"log_start":"145'462265","ondisk_log_start":"145'462265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3141,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 423s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6770331648,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":464411,"num_read_kb":1827628,"num_write":463292,"num_write_kb":1853168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8930,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:05.600349+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:46:05.600349+0000","last_peered":"2023-10-04T01:46:05.600349+0000","last_clean":"2023-10-04T01:46:05.600349+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:46:05.600349+0000","last_undegraded":"2023-10-04T01:46:05.600349+0000","last_fullsized":"2023-10-04T01:46:05.600349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'493304","reported_seq":973854,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:12.943374+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:46:12.943374+0000","last_peered":"2023-10-04T01:46:12.943374+0000","last_clean":"2023-10-04T01:46:12.943374+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:46:12.943374+0000","last_undegraded":"2023-10-04T01:46:12.943374+0000","last_fullsized":"2023-10-04T01:46:12.943374+0000","mapping_epoch":15,"log_start":"145'483265","ondisk_log_start":"145'483265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2863,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 424s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7083917312,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":485866,"num_read_kb":1912828,"num_write":484211,"num_write_kb":1936844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9498,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:05.579585+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:46:05.579585+0000","last_peered":"2023-10-04T01:46:05.579585+0000","last_clean":"2023-10-04T01:46:05.579585+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:46:05.579585+0000","last_undegraded":"2023-10-04T01:46:05.579585+0000","last_fullsized":"2023-10-04T01:46:05.579585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'469971","reported_seq":929133,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.942671+0000","last_change":"2023-10-04T01:46:05.448002+0000","last_active":"2023-10-04T01:46:10.942671+0000","last_peered":"2023-10-04T01:46:10.942671+0000","last_clean":"2023-10-04T01:46:10.942671+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:46:10.942671+0000","last_undegraded":"2023-10-04T01:46:10.942671+0000","last_fullsized":"2023-10-04T01:46:10.942671+0000","mapping_epoch":15,"log_start":"145'459963","ondisk_log_start":"145'459963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6766907411,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":464336,"num_read_kb":1827144,"num_write":463459,"num_write_kb":1853830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'17633","reported_seq":22946,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:10.699303+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:46:10.699303+0000","last_peered":"2023-10-04T01:46:10.699303+0000","last_clean":"2023-10-04T01:46:10.699303+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:46:10.699303+0000","last_undegraded":"2023-10-04T01:46:10.699303+0000","last_fullsized":"2023-10-04T01:46:10.699303+0000","mapping_epoch":12,"log_start":"79'7600","ondisk_log_start":"79'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":321,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:40.814041+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:45:40.814041+0000","last_peered":"2023-10-04T01:45:40.814041+0000","last_clean":"2023-10-04T01:45:40.814041+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:45:40.814041+0000","last_undegraded":"2023-10-04T01:45:40.814041+0000","last_fullsized":"2023-10-04T01:45:40.814041+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27537940499,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1889094,"num_read_kb":7434776,"num_write":1885217,"num_write_kb":7540862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15440142336,"data_stored":35127238694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40212,"ondisk_log_size":40212,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56798,"ondisk_log_size":56798,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9306304,"kb_used_data":7537912,"kb_used_omap":2381,"kb_used_meta":1766002,"kb_avail":85065536,"statfs":{"total":96636764160,"available":87107108864,"internally_reserved":0,"allocated":7718821888,"data_stored":17561100031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":1808386630},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.44800000000000001,"15min":0.39900000000000002},"min":{"1min":0.17899999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.3799999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51000000000000001,"15min":0.46000000000000002},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":2.419,"5min":5.319,"15min":5.319},"last":0.372}]},{"osd":1,"last update":"Wed Oct 4 01:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.47999999999999998},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57199999999999995,"5min":1.657,"15min":6.7709999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.48799999999999999,"15min":0.42599999999999999},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.61299999999999999,"5min":6.0890000000000004,"15min":7.415},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739244,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5554380,"kb_used_data":3709636,"kb_used_omap":997,"kb_used_meta":1843738,"kb_avail":88817460,"statfs":{"total":96636764160,"available":90949079040,"internally_reserved":0,"allocated":3798667264,"data_stored":8648443023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1887988123},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.97899999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47899999999999998,"15min":0.49399999999999999},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":1.0589999999999999,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.76700000000000002,"15min":0.72199999999999998},"min":{"1min":0.23799999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":1.363,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.60199999999999998,"15min":0.622},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.60399999999999998,"5min":10.531000000000001,"15min":14.503},"last":0.32400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5768684,"kb_used_data":3836324,"kb_used_omap":1018,"kb_used_meta":1931333,"kb_avail":88603156,"statfs":{"total":96636764160,"available":90729631744,"internally_reserved":0,"allocated":3928395776,"data_stored":8918976486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":1977685292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,9,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":1.1579999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":4.0419999999999998,"5min":5.7969999999999997,"15min":207.047},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52900000000000003,"15min":1.2230000000000001},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.217},"max":{"1min":2.137,"5min":6.069,"15min":207.00200000000001},"last":0.26400000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":0.84399999999999997,"15min":0.56699999999999995},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":14.118,"5min":14.118,"15min":14.118},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.244,"5min":0.59099999999999997,"15min":0.53700000000000003},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.051,"5min":14.051,"15min":14.051},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3925876736,"data_stored":8917852160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3795968000,"data_stored":8647540755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7718297600,"data_stored":17561845779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:46:14.265 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:14.265 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:14.265 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:14.265 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:14.266 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:46:19.456 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:46:19.446+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:46:19.456 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:46:19.450+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:46:19.464 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:46:19.458+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:46:19.465 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:46:19.458+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:46:34.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:46:34.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:46:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:46:34.917 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2346,"stamp":"2023-10-04T01:46:33.675341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27568460345,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2107137,"num_read_kb":7639009,"num_write":1948133,"num_write_kb":7626071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97241,"ondisk_log_size":97241,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20841220,"kb_used_data":15141612,"kb_used_omap":4397,"kb_used_meta":5695186,"kb_avail":262274300,"statfs":{"total":289910292480,"available":268568883200,"internally_reserved":0,"allocated":15505010688,"data_stored":35187645300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4503283,"internal_metadata":5831870733},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,4,7,8,4,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19079168,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4316,"num_read_kb":17208,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002109"},"pg_stats":[{"pgid":"2.7","version":"147'8064","reported_seq":11395,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:27.834328+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:46:27.834328+0000","last_peered":"2023-10-04T01:46:27.834328+0000","last_clean":"2023-10-04T01:46:27.834328+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:46:27.834328+0000","last_undegraded":"2023-10-04T01:46:27.834328+0000","last_fullsized":"2023-10-04T01:46:27.834328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8064,"log_dups_size":0,"ondisk_log_size":8064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'7283","reported_seq":10948,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:27.780951+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:46:27.780951+0000","last_peered":"2023-10-04T01:46:27.780951+0000","last_clean":"2023-10-04T01:46:27.780951+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:46:27.780951+0000","last_undegraded":"2023-10-04T01:46:27.780951+0000","last_fullsized":"2023-10-04T01:46:27.780951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7283,"log_dups_size":0,"ondisk_log_size":7283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'8207","reported_seq":11845,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:27.836783+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:46:27.836783+0000","last_peered":"2023-10-04T01:46:27.836783+0000","last_clean":"2023-10-04T01:46:27.836783+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:46:27.836783+0000","last_undegraded":"2023-10-04T01:46:27.836783+0000","last_fullsized":"2023-10-04T01:46:27.836783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8207,"log_dups_size":0,"ondisk_log_size":8207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'6638","reported_seq":11122,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:27.830080+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:46:27.830080+0000","last_peered":"2023-10-04T01:46:27.830080+0000","last_clean":"2023-10-04T01:46:27.830080+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:46:27.830080+0000","last_undegraded":"2023-10-04T01:46:27.830080+0000","last_fullsized":"2023-10-04T01:46:27.830080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6638,"log_dups_size":0,"ondisk_log_size":6638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'485294","reported_seq":956354,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:33.079217+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:46:33.079217+0000","last_peered":"2023-10-04T01:46:33.079217+0000","last_clean":"2023-10-04T01:46:33.079217+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:46:33.079217+0000","last_undegraded":"2023-10-04T01:46:33.079217+0000","last_fullsized":"2023-10-04T01:46:33.079217+0000","mapping_epoch":15,"log_start":"145'475273","ondisk_log_start":"145'475273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3202,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 448s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6924259328,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":476207,"num_read_kb":1874056,"num_write":476068,"num_write_kb":1904272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'16794","reported_seq":21894,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:33.030314+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:46:33.030314+0000","last_peered":"2023-10-04T01:46:33.030314+0000","last_clean":"2023-10-04T01:46:33.030314+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:46:33.030314+0000","last_undegraded":"2023-10-04T01:46:33.030314+0000","last_fullsized":"2023-10-04T01:46:33.030314+0000","mapping_epoch":12,"log_start":"77'6700","ondisk_log_start":"77'6700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'474202","reported_seq":935310,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:31.005845+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:46:31.005845+0000","last_peered":"2023-10-04T01:46:31.005845+0000","last_clean":"2023-10-04T01:46:31.005845+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:46:31.005845+0000","last_undegraded":"2023-10-04T01:46:31.005845+0000","last_fullsized":"2023-10-04T01:46:31.005845+0000","mapping_epoch":15,"log_start":"145'464165","ondisk_log_start":"145'464165","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3315,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 443s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6777409536,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":466157,"num_read_kb":1834584,"num_write":465014,"num_write_kb":1860056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8947,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:26.083139+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:46:26.083139+0000","last_peered":"2023-10-04T01:46:26.083139+0000","last_clean":"2023-10-04T01:46:26.083139+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:46:26.083139+0000","last_undegraded":"2023-10-04T01:46:26.083139+0000","last_fullsized":"2023-10-04T01:46:26.083139+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'495342","reported_seq":977607,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:33.073556+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:46:33.073556+0000","last_peered":"2023-10-04T01:46:33.073556+0000","last_clean":"2023-10-04T01:46:33.073556+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:46:33.073556+0000","last_undegraded":"2023-10-04T01:46:33.073556+0000","last_fullsized":"2023-10-04T01:46:33.073556+0000","mapping_epoch":15,"log_start":"145'485265","ondisk_log_start":"145'485265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2983,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 444s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7091892224,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":487568,"num_read_kb":1919608,"num_write":486151,"num_write_kb":1944604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9510,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:26.072810+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:46:26.072810+0000","last_peered":"2023-10-04T01:46:26.072810+0000","last_clean":"2023-10-04T01:46:26.072810+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:46:26.072810+0000","last_undegraded":"2023-10-04T01:46:26.072810+0000","last_fullsized":"2023-10-04T01:46:26.072810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'471768","reported_seq":932632,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:31.018632+0000","last_change":"2023-10-04T01:46:27.631391+0000","last_active":"2023-10-04T01:46:31.018632+0000","last_peered":"2023-10-04T01:46:31.018632+0000","last_clean":"2023-10-04T01:46:31.018632+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:46:31.018632+0000","last_undegraded":"2023-10-04T01:46:31.018632+0000","last_fullsized":"2023-10-04T01:46:31.018632+0000","mapping_epoch":15,"log_start":"145'461763","ondisk_log_start":"145'461763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6774292499,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":466031,"num_read_kb":1833908,"num_write":465256,"num_write_kb":1861018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'17685","reported_seq":23007,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:27.875442+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:46:27.875442+0000","last_peered":"2023-10-04T01:46:27.875442+0000","last_clean":"2023-10-04T01:46:27.875442+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:46:27.875442+0000","last_undegraded":"2023-10-04T01:46:27.875442+0000","last_fullsized":"2023-10-04T01:46:27.875442+0000","mapping_epoch":12,"log_start":"79'7600","ondisk_log_start":"79'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":321,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:40.814041+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:45:40.814041+0000","last_peered":"2023-10-04T01:45:40.814041+0000","last_clean":"2023-10-04T01:45:40.814041+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:45:40.814041+0000","last_undegraded":"2023-10-04T01:45:40.814041+0000","last_fullsized":"2023-10-04T01:45:40.814041+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27567853587,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1895963,"num_read_kb":7462156,"num_write":1892489,"num_write_kb":7569950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15499452416,"data_stored":35186548774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40140,"ondisk_log_size":40140,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57002,"ondisk_log_size":57002,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739249,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9403160,"kb_used_data":7566736,"kb_used_omap":2381,"kb_used_meta":1834034,"kb_avail":84968680,"statfs":{"total":96636764160,"available":87007928320,"internally_reserved":0,"allocated":7748337664,"data_stored":17590615807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":1878051398},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.44800000000000001,"15min":0.39900000000000002},"min":{"1min":0.17899999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.3799999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51000000000000001,"15min":0.46000000000000002},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":2.419,"5min":5.319,"15min":5.319},"last":0.436}]},{"osd":1,"last update":"Wed Oct 4 01:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.47999999999999998},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57199999999999995,"5min":1.657,"15min":6.7709999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.48799999999999999,"15min":0.42599999999999999},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.61299999999999999,"5min":6.0890000000000004,"15min":7.415},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359739248,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5600340,"kb_used_data":3723660,"kb_used_omap":997,"kb_used_meta":1875674,"kb_avail":88771500,"statfs":{"total":96636764160,"available":90902016000,"internally_reserved":0,"allocated":3813027840,"data_stored":8662803599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1920690587},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.97899999999999998,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47899999999999998,"15min":0.49399999999999999},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":1.0589999999999999,"5min":4.6429999999999998,"15min":6.1820000000000004},"last":3.266}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.76700000000000002,"15min":0.72199999999999998},"min":{"1min":0.23799999999999999,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":1.363,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.60199999999999998,"15min":0.622},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.60399999999999998,"5min":10.531000000000001,"15min":14.503},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5837720,"kb_used_data":3851216,"kb_used_omap":1018,"kb_used_meta":1985477,"kb_avail":88534120,"statfs":{"total":96636764160,"available":90658938880,"internally_reserved":0,"allocated":3943645184,"data_stored":8934225894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":2033128748},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,7,4,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:46:25 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":0.68100000000000005,"15min":1.2250000000000001},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":17.991,"5min":17.991,"15min":207.047},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.25,"5min":0.86199999999999999,"15min":1.343},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":17.957000000000001,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:45:37 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":0.84399999999999997,"15min":0.56699999999999995},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":14.118,"5min":14.118,"15min":14.118},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.244,"5min":0.59099999999999997,"15min":0.53700000000000003},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.051,"5min":14.051,"15min":14.051},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3941257216,"data_stored":8933232640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3810381824,"data_stored":8661954579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7747813376,"data_stored":17591361555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:46:34.919 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:34.920 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:34.920 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:34.920 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:34.920 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:46:48.272 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:46:48.265+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:46:48.273 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:46:48.265+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:46:48.273 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:46:48.265+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:46:54.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:46:55.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:46:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:46:55.514 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2356,"stamp":"2023-10-04T01:46:53.680011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27599331897,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2114203,"num_read_kb":7667181,"num_write":1955661,"num_write_kb":7656183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97312,"ondisk_log_size":97312,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21131820,"kb_used_data":15201748,"kb_used_omap":4397,"kb_used_meta":5925650,"kb_avail":261983700,"statfs":{"total":289910292480,"available":268271308800,"internally_reserved":0,"allocated":15566589952,"data_stored":35249224564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4503283,"internal_metadata":6067865869},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,1,8,16,15,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19218432,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4381,"num_read_kb":17472,"num_write":4685,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"2.7","version":"147'8094","reported_seq":11433,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:50.559840+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:46:50.559840+0000","last_peered":"2023-10-04T01:46:50.559840+0000","last_clean":"2023-10-04T01:46:50.559840+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:46:50.559840+0000","last_undegraded":"2023-10-04T01:46:50.559840+0000","last_fullsized":"2023-10-04T01:46:50.559840+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'7299","reported_seq":10968,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:50.560213+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:46:50.560213+0000","last_peered":"2023-10-04T01:46:50.560213+0000","last_clean":"2023-10-04T01:46:50.560213+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:46:50.560213+0000","last_undegraded":"2023-10-04T01:46:50.560213+0000","last_fullsized":"2023-10-04T01:46:50.560213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'8239","reported_seq":11885,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:50.559965+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:46:50.559965+0000","last_peered":"2023-10-04T01:46:50.559965+0000","last_clean":"2023-10-04T01:46:50.559965+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:46:50.559965+0000","last_undegraded":"2023-10-04T01:46:50.559965+0000","last_fullsized":"2023-10-04T01:46:50.559965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8239,"log_dups_size":0,"ondisk_log_size":8239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'6670","reported_seq":11163,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:51.116703+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:46:51.116703+0000","last_peered":"2023-10-04T01:46:51.116703+0000","last_clean":"2023-10-04T01:46:51.116703+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:46:51.116703+0000","last_undegraded":"2023-10-04T01:46:51.116703+0000","last_fullsized":"2023-10-04T01:46:51.116703+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6670,"log_dups_size":0,"ondisk_log_size":6670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"147'487275","reported_seq":960109,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:53.281013+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:46:53.281013+0000","last_peered":"2023-10-04T01:46:53.281013+0000","last_clean":"2023-10-04T01:46:53.281013+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:46:53.281013+0000","last_undegraded":"2023-10-04T01:46:53.281013+0000","last_fullsized":"2023-10-04T01:46:53.281013+0000","mapping_epoch":15,"log_start":"146'477273","ondisk_log_start":"146'477273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3288,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 469s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6932058112,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":477972,"num_read_kb":1881088,"num_write":477980,"num_write_kb":1911920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'16833","reported_seq":21946,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:51.658751+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:46:51.658751+0000","last_peered":"2023-10-04T01:46:51.658751+0000","last_clean":"2023-10-04T01:46:51.658751+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:46:51.658751+0000","last_undegraded":"2023-10-04T01:46:51.658751+0000","last_fullsized":"2023-10-04T01:46:51.658751+0000","mapping_epoch":12,"log_start":"79'6800","ondisk_log_start":"79'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'476147","reported_seq":939065,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:51.112314+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:46:51.112314+0000","last_peered":"2023-10-04T01:46:51.112314+0000","last_clean":"2023-10-04T01:46:51.112314+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:46:51.112314+0000","last_undegraded":"2023-10-04T01:46:51.112314+0000","last_fullsized":"2023-10-04T01:46:51.112314+0000","mapping_epoch":15,"log_start":"146'466065","ondisk_log_start":"146'466065","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3445,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 463s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6784966656,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":467955,"num_read_kb":1841748,"num_write":466856,"num_write_kb":1867424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8963,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:46.543741+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:46:46.543741+0000","last_peered":"2023-10-04T01:46:46.543741+0000","last_clean":"2023-10-04T01:46:46.543741+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:46:46.543741+0000","last_undegraded":"2023-10-04T01:46:46.543741+0000","last_fullsized":"2023-10-04T01:46:46.543741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"147'497341","reported_seq":981384,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:46:53.280815+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:46:53.280815+0000","last_peered":"2023-10-04T01:46:53.280815+0000","last_clean":"2023-10-04T01:46:53.280815+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:46:53.280815+0000","last_undegraded":"2023-10-04T01:46:53.280815+0000","last_fullsized":"2023-10-04T01:46:53.280815+0000","mapping_epoch":15,"log_start":"146'487265","ondisk_log_start":"146'487265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3079,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 465s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7099801600,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":489338,"num_read_kb":1926676,"num_write":488073,"num_write_kb":1952292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9522,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:46.538921+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:46:46.538921+0000","last_peered":"2023-10-04T01:46:46.538921+0000","last_clean":"2023-10-04T01:46:46.538921+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:46:46.538921+0000","last_undegraded":"2023-10-04T01:46:46.538921+0000","last_fullsized":"2023-10-04T01:46:46.538921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'473620","reported_seq":936228,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:51.092062+0000","last_change":"2023-10-04T01:46:50.787960+0000","last_active":"2023-10-04T01:46:51.092062+0000","last_peered":"2023-10-04T01:46:51.092062+0000","last_clean":"2023-10-04T01:46:51.092062+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:46:51.092062+0000","last_undegraded":"2023-10-04T01:46:51.092062+0000","last_fullsized":"2023-10-04T01:46:51.092062+0000","mapping_epoch":15,"log_start":"146'463563","ondisk_log_start":"146'463563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6781898771,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":467764,"num_read_kb":1840816,"num_write":467108,"num_write_kb":1868426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'17730","reported_seq":23065,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:50.584681+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:46:50.584681+0000","last_peered":"2023-10-04T01:46:50.584681+0000","last_clean":"2023-10-04T01:46:50.584681+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:46:50.584681+0000","last_undegraded":"2023-10-04T01:46:50.584681+0000","last_fullsized":"2023-10-04T01:46:50.584681+0000","mapping_epoch":12,"log_start":"81'7700","ondisk_log_start":"81'7700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":321,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:40.814041+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:45:40.814041+0000","last_peered":"2023-10-04T01:45:40.814041+0000","last_clean":"2023-10-04T01:45:40.814041+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:45:40.814041+0000","last_undegraded":"2023-10-04T01:45:40.814041+0000","last_fullsized":"2023-10-04T01:45:40.814041+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27598725139,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1903029,"num_read_kb":7490328,"num_write":1900017,"num_write_kb":7600062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15560994816,"data_stored":35248091174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40217,"ondisk_log_size":40217,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":56996,"ondisk_log_size":56996,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739253,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9555388,"kb_used_data":7596788,"kb_used_omap":2381,"kb_used_meta":1956210,"kb_avail":84816452,"statfs":{"total":96636764160,"available":86852046848,"internally_reserved":0,"allocated":7779110912,"data_stored":17621389055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":2003159622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.46200000000000002,"15min":0.41099999999999998},"min":{"1min":0.19,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.3259999999999996,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.52300000000000002,"15min":0.47199999999999998},"min":{"1min":0.25900000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":5.3600000000000003,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.30599999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":5.3929999999999998,"5min":5.3929999999999998,"15min":6.343},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.435},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":1.55,"5min":6.0890000000000004,"15min":7.415},"last":0.376}]}]},{"osd":1,"up_from":8,"seq":34359739252,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5669340,"kb_used_data":3738516,"kb_used_omap":997,"kb_used_meta":1929818,"kb_avail":88702500,"statfs":{"total":96636764160,"available":90831360000,"internally_reserved":0,"allocated":3828240384,"data_stored":8678016143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":1976134043},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,11],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.35999999999999999,"15min":0.375},"min":{"1min":0.217,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46600000000000003,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.19},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.48299999999999998},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.266,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.68700000000000006,"15min":0.71699999999999997},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.70499999999999996,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.307},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.65900000000000003,"5min":10.531000000000001,"15min":14.503},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5907092,"kb_used_data":3866444,"kb_used_omap":1018,"kb_used_meta":2039621,"kb_avail":88464748,"statfs":{"total":96636764160,"available":90587901952,"internally_reserved":0,"allocated":3959238656,"data_stored":8949819366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1043156,"internal_metadata":2088572204},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,3,14,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:46:25 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":0.68100000000000005,"15min":1.2250000000000001},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":17.991,"5min":17.991,"15min":207.047},"last":0.44},{"interface":"front","average":{"1min":2.25,"5min":0.86199999999999999,"15min":1.343},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":17.957000000000001,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.47599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.78700000000000003,"15min":0.56799999999999995},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":1.444,"5min":14.118,"15min":14.118},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.5269999999999999,"5min":14.051,"15min":14.051},"last":0.22600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3956957184,"data_stored":8948932608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3825512448,"data_stored":8677085203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7778525184,"data_stored":17622073363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:46:55.517 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:55.517 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:55.518 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:55.518 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:46:55.518 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:47:05.794 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.785+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:47:05.814 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.801+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:47:05.815 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.805+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:47:05.815 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.805+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:47:05.815 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.805+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:47:05.816 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:05.805+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:47:06.816 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.809+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:47:06.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:47:06.829 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:47:06.821+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:47:15.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:47:16.029 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:47:16.029 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:47:16.057 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2368,"stamp":"2023-10-04T01:47:15.683802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27629003321,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2121141,"num_read_kb":7694865,"num_write":1962864,"num_write_kb":7684995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97449,"ondisk_log_size":97449,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20813684,"kb_used_data":15266200,"kb_used_omap":4399,"kb_used_meta":5543056,"kb_avail":262301836,"statfs":{"total":289910292480,"available":268597080064,"internally_reserved":0,"allocated":15632588800,"data_stored":35315215560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4504963,"internal_metadata":5676089981},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,6,8,9,11,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14991360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3534,"num_read_kb":14104,"num_write":3644,"num_write_kb":14576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001705"},"pg_stats":[{"pgid":"2.7","version":"147'8116","reported_seq":11463,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:10.599474+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:47:10.599474+0000","last_peered":"2023-10-04T01:47:10.599474+0000","last_clean":"2023-10-04T01:47:10.599474+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:47:10.599474+0000","last_undegraded":"2023-10-04T01:47:10.599474+0000","last_fullsized":"2023-10-04T01:47:10.599474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8116,"log_dups_size":0,"ondisk_log_size":8116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'7309","reported_seq":10983,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:10.599634+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:47:10.599634+0000","last_peered":"2023-10-04T01:47:10.599634+0000","last_clean":"2023-10-04T01:47:10.599634+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:47:10.599634+0000","last_undegraded":"2023-10-04T01:47:10.599634+0000","last_fullsized":"2023-10-04T01:47:10.599634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'8259","reported_seq":11914,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:10.602167+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:47:10.602167+0000","last_peered":"2023-10-04T01:47:10.602167+0000","last_clean":"2023-10-04T01:47:10.602167+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:47:10.602167+0000","last_undegraded":"2023-10-04T01:47:10.602167+0000","last_fullsized":"2023-10-04T01:47:10.602167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8259,"log_dups_size":0,"ondisk_log_size":8259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'6690","reported_seq":11191,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:10.600508+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:47:10.600508+0000","last_peered":"2023-10-04T01:47:10.600508+0000","last_clean":"2023-10-04T01:47:10.600508+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:47:10.600508+0000","last_undegraded":"2023-10-04T01:47:10.600508+0000","last_fullsized":"2023-10-04T01:47:10.600508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'489165","reported_seq":963729,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:13.410991+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:47:13.410991+0000","last_peered":"2023-10-04T01:47:13.410991+0000","last_clean":"2023-10-04T01:47:13.410991+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:47:13.410991+0000","last_undegraded":"2023-10-04T01:47:13.410991+0000","last_fullsized":"2023-10-04T01:47:13.410991+0000","mapping_epoch":15,"log_start":"146'479073","ondisk_log_start":"146'479073","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3478,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 489s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6939222016,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":479692,"num_read_kb":1887956,"num_write":479716,"num_write_kb":1918864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"147'16867","reported_seq":21993,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:12.222922+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:47:12.222922+0000","last_peered":"2023-10-04T01:47:12.222922+0000","last_clean":"2023-10-04T01:47:12.222922+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:47:12.222922+0000","last_undegraded":"2023-10-04T01:47:12.222922+0000","last_fullsized":"2023-10-04T01:47:12.222922+0000","mapping_epoch":12,"log_start":"79'6800","ondisk_log_start":"79'6800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"147'478086","reported_seq":942737,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:11.234267+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:47:11.234267+0000","last_peered":"2023-10-04T01:47:11.234267+0000","last_clean":"2023-10-04T01:47:11.234267+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:47:11.234267+0000","last_undegraded":"2023-10-04T01:47:11.234267+0000","last_fullsized":"2023-10-04T01:47:11.234267+0000","mapping_epoch":15,"log_start":"146'468066","ondisk_log_start":"146'468066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3628,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 483s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6792294400,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":469671,"num_read_kb":1848596,"num_write":468642,"num_write_kb":1874568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":8984,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:12.140800+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:47:12.140800+0000","last_peered":"2023-10-04T01:47:12.140800+0000","last_clean":"2023-10-04T01:47:12.140800+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:47:12.140800+0000","last_undegraded":"2023-10-04T01:47:12.140800+0000","last_fullsized":"2023-10-04T01:47:12.140800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'499330","reported_seq":985195,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:13.404554+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:47:13.404554+0000","last_peered":"2023-10-04T01:47:13.404554+0000","last_clean":"2023-10-04T01:47:13.404554+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:47:13.404554+0000","last_undegraded":"2023-10-04T01:47:13.404554+0000","last_fullsized":"2023-10-04T01:47:13.404554+0000","mapping_epoch":15,"log_start":"146'489265","ondisk_log_start":"146'489265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3189,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 485s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7107620864,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":491150,"num_read_kb":1933900,"num_write":489972,"num_write_kb":1959888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9538,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:12.148423+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:47:12.148423+0000","last_peered":"2023-10-04T01:47:12.148423+0000","last_clean":"2023-10-04T01:47:12.148423+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:47:12.148423+0000","last_undegraded":"2023-10-04T01:47:12.148423+0000","last_fullsized":"2023-10-04T01:47:12.148423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"147'475402","reported_seq":939703,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:11.253190+0000","last_change":"2023-10-04T01:47:06.675862+0000","last_active":"2023-10-04T01:47:11.253190+0000","last_peered":"2023-10-04T01:47:11.253190+0000","last_clean":"2023-10-04T01:47:11.253190+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:47:11.253190+0000","last_undegraded":"2023-10-04T01:47:11.253190+0000","last_fullsized":"2023-10-04T01:47:11.253190+0000","mapping_epoch":15,"log_start":"146'465363","ondisk_log_start":"146'465363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6789259283,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":469454,"num_read_kb":1847560,"num_write":468890,"num_write_kb":1875554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"147'17762","reported_seq":23109,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:47:10.645362+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:47:10.645362+0000","last_peered":"2023-10-04T01:47:10.645362+0000","last_clean":"2023-10-04T01:47:10.645362+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:47:10.645362+0000","last_undegraded":"2023-10-04T01:47:10.645362+0000","last_fullsized":"2023-10-04T01:47:10.645362+0000","mapping_epoch":12,"log_start":"81'7700","ondisk_log_start":"81'7700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":321,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-04T01:45:40.814041+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:45:40.814041+0000","last_peered":"2023-10-04T01:45:40.814041+0000","last_clean":"2023-10-04T01:45:40.814041+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:45:40.814041+0000","last_undegraded":"2023-10-04T01:45:40.814041+0000","last_fullsized":"2023-10-04T01:45:40.814041+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27628396563,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1909967,"num_read_kb":7518012,"num_write":1907220,"num_write_kb":7628874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15627644928,"data_stored":35314741286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40216,"ondisk_log_size":40216,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57134,"ondisk_log_size":57134,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739258,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9714000,"kb_used_data":7632900,"kb_used_omap":2381,"kb_used_meta":2078706,"kb_avail":84657840,"statfs":{"total":96636764160,"available":86689628160,"internally_reserved":0,"allocated":7816089600,"data_stored":17658359891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":2128595526},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.46200000000000002,"15min":0.41099999999999998},"min":{"1min":0.19,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.3259999999999996,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.52300000000000002,"15min":0.47199999999999998},"min":{"1min":0.25900000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":5.3600000000000003,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":5.3929999999999998,"5min":5.3929999999999998,"15min":6.343},"last":4.4160000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.435},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":1.55,"5min":6.0890000000000004,"15min":7.415},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739256,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5737472,"kb_used_data":3752504,"kb_used_omap":997,"kb_used_meta":1983962,"kb_avail":88634368,"statfs":{"total":96636764160,"available":90761592832,"internally_reserved":0,"allocated":3842564096,"data_stored":8692339855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021541,"internal_metadata":2031577499},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.35999999999999999,"15min":0.375},"min":{"1min":0.217,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46600000000000003,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.48299999999999998},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.266,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.20300000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.68700000000000006,"15min":0.71699999999999997},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.70499999999999996,"5min":10.494999999999999,"15min":14.467000000000001},"last":1.222},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.65900000000000003,"5min":10.531000000000001,"15min":14.503},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5362212,"kb_used_data":3880796,"kb_used_omap":1020,"kb_used_meta":1480387,"kb_avail":89009628,"statfs":{"total":96636764160,"available":91145859072,"internally_reserved":0,"allocated":3973935104,"data_stored":8964515814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1515916956},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,3,6,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:46:25 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":0.68100000000000005,"15min":1.2250000000000001},"min":{"1min":0.216,"5min":0.188,"15min":0.188},"max":{"1min":17.991,"5min":17.991,"15min":207.047},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.25,"5min":0.86199999999999999,"15min":1.343},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":17.957000000000001,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.317}]},{"osd":2,"last update":"Wed Oct 4 01:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.78700000000000003,"15min":0.56799999999999995},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":1.444,"5min":14.118,"15min":14.118},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.5269999999999999,"5min":14.051,"15min":14.051},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3971858432,"data_stored":8963833856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3840126976,"data_stored":8691699731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7815659520,"data_stored":17659207699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:47:16.059 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:16.060 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:16.060 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:16.060 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:16.060 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:47:36.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:47:36.604 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:47:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:47:36.667 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2378,"stamp":"2023-10-04T01:47:35.688058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27658056249,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2127816,"num_read_kb":7721513,"num_write":1969930,"num_write_kb":7713259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97381,"ondisk_log_size":97381,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20443712,"kb_used_data":15324328,"kb_used_omap":4401,"kb_used_meta":5114958,"kb_avail":262671808,"statfs":{"total":289910292480,"available":268975931392,"internally_reserved":0,"allocated":15692111872,"data_stored":35374722928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4506642,"internal_metadata":5237717998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,8,6,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14147584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3291,"num_read_kb":13140,"num_write":3448,"num_write_kb":13792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002194"},"pg_stats":[{"pgid":"2.7","version":"148'8142","reported_seq":11499,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:30.023759+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:47:30.023759+0000","last_peered":"2023-10-04T01:47:30.023759+0000","last_clean":"2023-10-04T01:47:30.023759+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:47:30.023759+0000","last_undegraded":"2023-10-04T01:47:30.023759+0000","last_fullsized":"2023-10-04T01:47:30.023759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'7322","reported_seq":11001,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:27.622900+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:47:27.622900+0000","last_peered":"2023-10-04T01:47:27.622900+0000","last_clean":"2023-10-04T01:47:27.622900+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:47:27.622900+0000","last_undegraded":"2023-10-04T01:47:27.622900+0000","last_fullsized":"2023-10-04T01:47:27.622900+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7322,"log_dups_size":0,"ondisk_log_size":7322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'8285","reported_seq":11949,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:30.014641+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:47:30.014641+0000","last_peered":"2023-10-04T01:47:30.014641+0000","last_clean":"2023-10-04T01:47:30.014641+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:47:30.014641+0000","last_undegraded":"2023-10-04T01:47:30.014641+0000","last_fullsized":"2023-10-04T01:47:30.014641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'6716","reported_seq":11226,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:30.015912+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:47:30.015912+0000","last_peered":"2023-10-04T01:47:30.015912+0000","last_clean":"2023-10-04T01:47:30.015912+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:47:30.015912+0000","last_undegraded":"2023-10-04T01:47:30.015912+0000","last_fullsized":"2023-10-04T01:47:30.015912+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'491016","reported_seq":967361,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:33.583378+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:47:33.583378+0000","last_peered":"2023-10-04T01:47:33.583378+0000","last_clean":"2023-10-04T01:47:33.583378+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:47:33.583378+0000","last_undegraded":"2023-10-04T01:47:33.583378+0000","last_fullsized":"2023-10-04T01:47:33.583378+0000","mapping_epoch":15,"log_start":"146'480973","ondisk_log_start":"146'480973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3647,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 509s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6946287616,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":481456,"num_read_kb":1895000,"num_write":481432,"num_write_kb":1925728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'16915","reported_seq":22053,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:32.619053+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:47:32.619053+0000","last_peered":"2023-10-04T01:47:32.619053+0000","last_clean":"2023-10-04T01:47:32.619053+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:47:32.619053+0000","last_undegraded":"2023-10-04T01:47:32.619053+0000","last_fullsized":"2023-10-04T01:47:32.619053+0000","mapping_epoch":12,"log_start":"79'6900","ondisk_log_start":"79'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'479999","reported_seq":946298,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:31.312268+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:47:31.312268+0000","last_peered":"2023-10-04T01:47:31.312268+0000","last_clean":"2023-10-04T01:47:31.312268+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:47:31.312268+0000","last_undegraded":"2023-10-04T01:47:31.312268+0000","last_fullsized":"2023-10-04T01:47:31.312268+0000","mapping_epoch":15,"log_start":"146'469966","ondisk_log_start":"146'469966","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3787,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 504s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6799605760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":471305,"num_read_kb":1855104,"num_write":470423,"num_write_kb":1881692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9000,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:32.685070+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:47:32.685070+0000","last_peered":"2023-10-04T01:47:32.685070+0000","last_clean":"2023-10-04T01:47:32.685070+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:47:32.685070+0000","last_undegraded":"2023-10-04T01:47:32.685070+0000","last_fullsized":"2023-10-04T01:47:32.685070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'501282","reported_seq":988832,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:33.580851+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:47:33.580851+0000","last_peered":"2023-10-04T01:47:33.580851+0000","last_clean":"2023-10-04T01:47:33.580851+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:47:33.580851+0000","last_undegraded":"2023-10-04T01:47:33.580851+0000","last_fullsized":"2023-10-04T01:47:33.580851+0000","mapping_epoch":15,"log_start":"146'491265","ondisk_log_start":"146'491265","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3340,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 505s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7115145216,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":492825,"num_read_kb":1940592,"num_write":491803,"num_write_kb":1967212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9550,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:32.619141+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:47:32.619141+0000","last_peered":"2023-10-04T01:47:32.619141+0000","last_clean":"2023-10-04T01:47:32.619141+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:47:32.619141+0000","last_undegraded":"2023-10-04T01:47:32.619141+0000","last_fullsized":"2023-10-04T01:47:32.619141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'477140","reported_seq":943051,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:31.291438+0000","last_change":"2023-10-04T01:47:23.901119+0000","last_active":"2023-10-04T01:47:31.291438+0000","last_peered":"2023-10-04T01:47:31.291438+0000","last_clean":"2023-10-04T01:47:31.291438+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:47:31.291438+0000","last_undegraded":"2023-10-04T01:47:31.291438+0000","last_fullsized":"2023-10-04T01:47:31.291438+0000","mapping_epoch":15,"log_start":"146'467063","ondisk_log_start":"146'467063","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6796410899,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":471056,"num_read_kb":1853964,"num_write":470628,"num_write_kb":1882506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'17801","reported_seq":23161,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:30.051325+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:47:30.051325+0000","last_peered":"2023-10-04T01:47:30.051325+0000","last_clean":"2023-10-04T01:47:30.051325+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:47:30.051325+0000","last_undegraded":"2023-10-04T01:47:30.051325+0000","last_fullsized":"2023-10-04T01:47:30.051325+0000","mapping_epoch":12,"log_start":"81'7800","ondisk_log_start":"81'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":322,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-04T01:46:03.265806+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:46:03.265806+0000","last_peered":"2023-10-04T01:46:03.265806+0000","last_clean":"2023-10-04T01:46:03.265806+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:46:03.265806+0000","last_undegraded":"2023-10-04T01:46:03.265806+0000","last_fullsized":"2023-10-04T01:46:03.265806+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27657449491,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1916642,"num_read_kb":7544660,"num_write":1914286,"num_write_kb":7657138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15685611520,"data_stored":35372707878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40170,"ondisk_log_size":40170,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57112,"ondisk_log_size":57112,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739262,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9839956,"kb_used_data":7662348,"kb_used_omap":2381,"kb_used_meta":2175218,"kb_avail":84531884,"statfs":{"total":96636764160,"available":86560649216,"internally_reserved":0,"allocated":7846244352,"data_stored":17688514643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2438586,"internal_metadata":2227423814},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.46200000000000002,"15min":0.41099999999999998},"min":{"1min":0.19,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.3259999999999996,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.52300000000000002,"15min":0.47199999999999998},"min":{"1min":0.25900000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":5.3600000000000003,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":5.3929999999999998,"5min":5.3929999999999998,"15min":6.343},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47699999999999998,"15min":0.435},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":1.55,"5min":6.0890000000000004,"15min":7.415},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5191132,"kb_used_data":3766868,"kb_used_omap":999,"kb_used_meta":1423256,"kb_avail":89180708,"statfs":{"total":96636764160,"available":91321044992,"internally_reserved":0,"allocated":3857272832,"data_stored":8707040739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1457414924},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.35999999999999999,"15min":0.375},"min":{"1min":0.217,"5min":0.189,"15min":0.17899999999999999},"max":{"1min":0.46600000000000003,"5min":3.1869999999999998,"15min":5.7240000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.505,"15min":0.48299999999999998},"min":{"1min":0.23200000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":3.266,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.31}]},{"osd":2,"last update":"Wed Oct 4 01:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.68700000000000006,"15min":0.71699999999999997},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":0.70499999999999996,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.65900000000000003,"5min":10.531000000000001,"15min":14.503},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5412624,"kb_used_data":3895112,"kb_used_omap":1020,"kb_used_meta":1516483,"kb_avail":88959216,"statfs":{"total":96636764160,"available":91094237184,"internally_reserved":0,"allocated":3988594688,"data_stored":8979167546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1552879260},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.68500000000000005,"15min":1.23},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.63800000000000001,"5min":17.991,"15min":207.047},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.86399999999999999,"15min":1.3440000000000001},"min":{"1min":0.24099999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.60999999999999999,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:46:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.78700000000000003,"15min":0.56799999999999995},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":1.444,"5min":14.118,"15min":14.118},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.5269999999999999,"5min":14.051,"15min":14.051},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3986370560,"data_stored":8978345984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3854548992,"data_stored":8706121747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7844691968,"data_stored":17688240147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:47:36.669 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:36.670 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:36.670 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:36.670 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:36.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:47:37.225 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:47:37.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:47:38.101 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:47:38.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:47:38.796 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:47:38.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:47:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:47:39.397 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:47:59.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:47:59.868 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:47:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:47:59.905 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2390,"stamp":"2023-10-04T01:47:59.692142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27696534073,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2137012,"num_read_kb":7758201,"num_write":1979291,"num_write_kb":7750703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97653,"ondisk_log_size":97653,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20125392,"kb_used_data":15391416,"kb_used_omap":4402,"kb_used_meta":4729549,"kb_avail":262990128,"statfs":{"total":289910292480,"available":269301891072,"internally_reserved":0,"allocated":15760809984,"data_stored":35443421040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":4843058526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,4,5,11,6,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19451904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4671,"num_read_kb":18636,"num_write":4738,"num_write_kb":18952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001994"},"pg_stats":[{"pgid":"2.7","version":"148'8184","reported_seq":11551,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:56.121785+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:47:56.121785+0000","last_peered":"2023-10-04T01:47:56.121785+0000","last_clean":"2023-10-04T01:47:56.121785+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:47:56.121785+0000","last_undegraded":"2023-10-04T01:47:56.121785+0000","last_fullsized":"2023-10-04T01:47:56.121785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8184,"log_dups_size":0,"ondisk_log_size":8184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'7343","reported_seq":11027,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:53.098795+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:47:53.098795+0000","last_peered":"2023-10-04T01:47:53.098795+0000","last_clean":"2023-10-04T01:47:53.098795+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:47:53.098795+0000","last_undegraded":"2023-10-04T01:47:53.098795+0000","last_fullsized":"2023-10-04T01:47:53.098795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7343,"log_dups_size":0,"ondisk_log_size":7343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'8327","reported_seq":12001,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:56.120942+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:47:56.120942+0000","last_peered":"2023-10-04T01:47:56.120942+0000","last_clean":"2023-10-04T01:47:56.120942+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:47:56.120942+0000","last_undegraded":"2023-10-04T01:47:56.120942+0000","last_fullsized":"2023-10-04T01:47:56.120942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8327,"log_dups_size":0,"ondisk_log_size":8327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'6758","reported_seq":11278,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:56.118330+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:47:56.118330+0000","last_peered":"2023-10-04T01:47:56.118330+0000","last_clean":"2023-10-04T01:47:56.118330+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:47:56.118330+0000","last_undegraded":"2023-10-04T01:47:56.118330+0000","last_fullsized":"2023-10-04T01:47:56.118330+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'493460","reported_seq":972177,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:58.770002+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:47:58.770002+0000","last_peered":"2023-10-04T01:47:58.770002+0000","last_clean":"2023-10-04T01:47:58.770002+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:47:58.770002+0000","last_undegraded":"2023-10-04T01:47:58.770002+0000","last_fullsized":"2023-10-04T01:47:58.770002+0000","mapping_epoch":15,"log_start":"147'483373","ondisk_log_start":"147'483373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3788,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 534s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6955905024,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":483811,"num_read_kb":1904388,"num_write":483764,"num_write_kb":1935056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'16940","reported_seq":22093,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:58.218747+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:47:58.218747+0000","last_peered":"2023-10-04T01:47:58.218747+0000","last_clean":"2023-10-04T01:47:58.218747+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:47:58.218747+0000","last_undegraded":"2023-10-04T01:47:58.218747+0000","last_fullsized":"2023-10-04T01:47:58.218747+0000","mapping_epoch":12,"log_start":"79'6900","ondisk_log_start":"79'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'482412","reported_seq":951001,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:56.435470+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:47:56.435470+0000","last_peered":"2023-10-04T01:47:56.435470+0000","last_clean":"2023-10-04T01:47:56.435470+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:47:56.435470+0000","last_undegraded":"2023-10-04T01:47:56.435470+0000","last_fullsized":"2023-10-04T01:47:56.435470+0000","mapping_epoch":15,"log_start":"147'472366","ondisk_log_start":"147'472366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3903,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 529s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6809128960,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":473585,"num_read_kb":1864208,"num_write":472743,"num_write_kb":1890972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9016,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:53.104909+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:47:53.104909+0000","last_peered":"2023-10-04T01:47:53.104909+0000","last_clean":"2023-10-04T01:47:53.104909+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:47:53.104909+0000","last_undegraded":"2023-10-04T01:47:53.104909+0000","last_fullsized":"2023-10-04T01:47:53.104909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'503767","reported_seq":993692,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:47:58.781987+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:47:58.781987+0000","last_peered":"2023-10-04T01:47:58.781987+0000","last_clean":"2023-10-04T01:47:58.781987+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:47:58.781987+0000","last_undegraded":"2023-10-04T01:47:58.781987+0000","last_fullsized":"2023-10-04T01:47:58.781987+0000","mapping_epoch":15,"log_start":"147'493765","ondisk_log_start":"147'493765","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3425,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 530s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7125016576,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":495193,"num_read_kb":1950032,"num_write":494217,"num_write_kb":1976868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9563,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:53.108059+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:47:53.108059+0000","last_peered":"2023-10-04T01:47:53.108059+0000","last_clean":"2023-10-04T01:47:53.108059+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:47:53.108059+0000","last_undegraded":"2023-10-04T01:47:53.108059+0000","last_fullsized":"2023-10-04T01:47:53.108059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'479435","reported_seq":947546,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:56.421272+0000","last_change":"2023-10-04T01:47:55.114037+0000","last_active":"2023-10-04T01:47:56.421272+0000","last_peered":"2023-10-04T01:47:56.421272+0000","last_clean":"2023-10-04T01:47:56.421272+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:47:56.421272+0000","last_undegraded":"2023-10-04T01:47:56.421272+0000","last_fullsized":"2023-10-04T01:47:56.421272+0000","mapping_epoch":15,"log_start":"147'469363","ondisk_log_start":"147'469363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6805876755,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":473249,"num_read_kb":1862720,"num_write":472923,"num_write_kb":1891686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'17864","reported_seq":23239,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:56.157959+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:47:56.157959+0000","last_peered":"2023-10-04T01:47:56.157959+0000","last_clean":"2023-10-04T01:47:56.157959+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:47:56.157959+0000","last_undegraded":"2023-10-04T01:47:56.157959+0000","last_fullsized":"2023-10-04T01:47:56.157959+0000","mapping_epoch":12,"log_start":"81'7800","ondisk_log_start":"81'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":323,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:13.345237+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:47:13.345237+0000","last_peered":"2023-10-04T01:47:13.345237+0000","last_clean":"2023-10-04T01:47:13.345237+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:47:13.345237+0000","last_undegraded":"2023-10-04T01:47:13.345237+0000","last_fullsized":"2023-10-04T01:47:13.345237+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27695927315,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1925838,"num_read_kb":7581348,"num_write":1923647,"num_write_kb":7694582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15754420224,"data_stored":35441516582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57347,"ondisk_log_size":57347,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739266,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9424428,"kb_used_data":7691940,"kb_used_omap":2383,"kb_used_meta":1730096,"kb_avail":84947412,"statfs":{"total":96636764160,"available":86986149888,"internally_reserved":0,"allocated":7876546560,"data_stored":17718816851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1771619254},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.48399999999999999,"15min":0.42399999999999999},"min":{"1min":0.20499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.702,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.48399999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":3.7440000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.42899999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.54700000000000004,"15min":0.46800000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4160000000000004,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.48399999999999999,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":3.5270000000000001,"5min":4.0579999999999998,"15min":7.415},"last":0.36599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739265,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5229080,"kb_used_data":3785360,"kb_used_omap":999,"kb_used_meta":1442712,"kb_avail":89142760,"statfs":{"total":96636764160,"available":91282186240,"internally_reserved":0,"allocated":3876208640,"data_stored":8725976547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1477337868},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.32500000000000001,"15min":0.374},"min":{"1min":0.19,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":0.49099999999999999,"5min":0.97899999999999998,"15min":5.7240000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.46600000000000003,"15min":0.48299999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.5,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.68700000000000006},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":2.7269999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.052,"5min":0.68100000000000005,"15min":0.62},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":6.1050000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5471884,"kb_used_data":3914116,"kb_used_omap":1020,"kb_used_meta":1556739,"kb_avail":88899956,"statfs":{"total":96636764160,"available":91033554944,"internally_reserved":0,"allocated":4008054784,"data_stored":8998627642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1594101404},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,6,3,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.68500000000000005,"15min":1.23},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.63800000000000001,"5min":17.991,"15min":207.047},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.86399999999999999,"15min":1.3440000000000001},"min":{"1min":0.24099999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.60999999999999999,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.47499999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:47:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.77900000000000003,"15min":0.56899999999999995},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.67500000000000004,"5min":14.118,"15min":14.118},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.7000000000000002,"5min":14.051,"15min":14.051},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4005818368,"data_stored":8997793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3873443840,"data_stored":8725016595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7875158016,"data_stored":17718706195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:47:59.907 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:59.907 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:59.907 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:59.908 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:47:59.908 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:48:18.919 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:48:18.911+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:48:18.920 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:48:18.911+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:48:18.920 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:48:18.911+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:48:19.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:48:20.478 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:48:20.479 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:48:20.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2400,"stamp":"2023-10-04T01:48:19.695485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27727094329,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2144403,"num_read_kb":7787693,"num_write":1986751,"num_write_kb":7780543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97788,"ondisk_log_size":97788,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20415560,"kb_used_data":15451120,"kb_used_omap":4402,"kb_used_meta":4960013,"kb_avail":262699960,"statfs":{"total":289910292480,"available":269004759040,"internally_reserved":0,"allocated":15821946880,"data_stored":35504557936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5079053662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,9,5,9,17,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19087360,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4621,"num_read_kb":18440,"num_write":4657,"num_write_kb":18628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002016"},"pg_stats":[{"pgid":"2.7","version":"148'8208","reported_seq":11583,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:15.330960+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:48:15.330960+0000","last_peered":"2023-10-04T01:48:15.330960+0000","last_clean":"2023-10-04T01:48:15.330960+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:48:15.330960+0000","last_undegraded":"2023-10-04T01:48:15.330960+0000","last_fullsized":"2023-10-04T01:48:15.330960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"148'7355","reported_seq":11043,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:14.943389+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:48:14.943389+0000","last_peered":"2023-10-04T01:48:14.943389+0000","last_clean":"2023-10-04T01:48:14.943389+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:48:14.943389+0000","last_undegraded":"2023-10-04T01:48:14.943389+0000","last_fullsized":"2023-10-04T01:48:14.943389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"148'8351","reported_seq":12033,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:15.341329+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:48:15.341329+0000","last_peered":"2023-10-04T01:48:15.341329+0000","last_clean":"2023-10-04T01:48:15.341329+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:48:15.341329+0000","last_undegraded":"2023-10-04T01:48:15.341329+0000","last_fullsized":"2023-10-04T01:48:15.341329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"148'6782","reported_seq":11310,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:15.334294+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:48:15.334294+0000","last_peered":"2023-10-04T01:48:15.334294+0000","last_clean":"2023-10-04T01:48:15.334294+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:48:15.334294+0000","last_undegraded":"2023-10-04T01:48:15.334294+0000","last_fullsized":"2023-10-04T01:48:15.334294+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6782,"log_dups_size":0,"ondisk_log_size":6782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"148'495414","reported_seq":976025,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:18.902965+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:48:18.902965+0000","last_peered":"2023-10-04T01:48:18.902965+0000","last_clean":"2023-10-04T01:48:18.902965+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:48:18.902965+0000","last_undegraded":"2023-10-04T01:48:18.902965+0000","last_fullsized":"2023-10-04T01:48:18.902965+0000","mapping_epoch":15,"log_start":"147'485373","ondisk_log_start":"147'485373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3908,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 554s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6963539968,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":485695,"num_read_kb":1911908,"num_write":485621,"num_write_kb":1942484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"148'16967","reported_seq":22132,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:18.702819+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:48:18.702819+0000","last_peered":"2023-10-04T01:48:18.702819+0000","last_clean":"2023-10-04T01:48:18.702819+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:48:18.702819+0000","last_undegraded":"2023-10-04T01:48:18.702819+0000","last_fullsized":"2023-10-04T01:48:18.702819+0000","mapping_epoch":12,"log_start":"79'6900","ondisk_log_start":"79'6900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"148'484385","reported_seq":954747,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:16.570016+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:48:16.570016+0000","last_peered":"2023-10-04T01:48:16.570016+0000","last_clean":"2023-10-04T01:48:16.570016+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:48:16.570016+0000","last_undegraded":"2023-10-04T01:48:16.570016+0000","last_fullsized":"2023-10-04T01:48:16.570016+0000","mapping_epoch":15,"log_start":"147'474366","ondisk_log_start":"147'474366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4054,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 549s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6816702464,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":475346,"num_read_kb":1871232,"num_write":474591,"num_write_kb":1898364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9032,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:13.581544+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:48:13.581544+0000","last_peered":"2023-10-04T01:48:13.581544+0000","last_clean":"2023-10-04T01:48:13.581544+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:48:13.581544+0000","last_undegraded":"2023-10-04T01:48:13.581544+0000","last_fullsized":"2023-10-04T01:48:13.581544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"148'505742","reported_seq":997579,"reported_epoch":148,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:18.900884+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:48:18.900884+0000","last_peered":"2023-10-04T01:48:18.900884+0000","last_clean":"2023-10-04T01:48:18.900884+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:48:18.900884+0000","last_undegraded":"2023-10-04T01:48:18.900884+0000","last_fullsized":"2023-10-04T01:48:18.900884+0000","mapping_epoch":15,"log_start":"147'495665","ondisk_log_start":"147'495665","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3554,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 550s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7132647424,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":497085,"num_read_kb":1957576,"num_write":496086,"num_write_kb":1984344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9575,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:13.603920+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:48:13.603920+0000","last_peered":"2023-10-04T01:48:13.603920+0000","last_clean":"2023-10-04T01:48:13.603920+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:48:13.603920+0000","last_undegraded":"2023-10-04T01:48:13.603920+0000","last_fullsized":"2023-10-04T01:48:13.603920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"148'481321","reported_seq":951291,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:16.563012+0000","last_change":"2023-10-04T01:48:05.372043+0000","last_active":"2023-10-04T01:48:16.563012+0000","last_peered":"2023-10-04T01:48:16.563012+0000","last_clean":"2023-10-04T01:48:16.563012+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:48:16.563012+0000","last_undegraded":"2023-10-04T01:48:16.563012+0000","last_fullsized":"2023-10-04T01:48:16.563012+0000","mapping_epoch":15,"log_start":"147'471263","ondisk_log_start":"147'471263","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6813597715,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":475103,"num_read_kb":1870124,"num_write":474809,"num_write_kb":1899230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"148'17900","reported_seq":23287,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:48:15.384155+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:48:15.384155+0000","last_peered":"2023-10-04T01:48:15.384155+0000","last_clean":"2023-10-04T01:48:15.384155+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:48:15.384155+0000","last_undegraded":"2023-10-04T01:48:15.384155+0000","last_fullsized":"2023-10-04T01:48:15.384155+0000","mapping_epoch":12,"log_start":"81'7800","ondisk_log_start":"81'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":323,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:13.345237+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:47:13.345237+0000","last_peered":"2023-10-04T01:47:13.345237+0000","last_clean":"2023-10-04T01:47:13.345237+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:47:13.345237+0000","last_undegraded":"2023-10-04T01:47:13.345237+0000","last_fullsized":"2023-10-04T01:47:13.345237+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27726487571,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1933229,"num_read_kb":7610840,"num_write":1931107,"num_write_kb":7724422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15815544832,"data_stored":35502641190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40195,"ondisk_log_size":40195,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57494,"ondisk_log_size":57494,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739270,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9576332,"kb_used_data":7721668,"kb_used_omap":2383,"kb_used_meta":1852272,"kb_avail":84795508,"statfs":{"total":96636764160,"available":86830600192,"internally_reserved":0,"allocated":7906988032,"data_stored":17749258323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1896727478},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.48399999999999999,"15min":0.42399999999999999},"min":{"1min":0.20499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.702,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.48399999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":3.7440000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.33200000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.54700000000000004,"15min":0.46800000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4160000000000004,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.48399999999999999,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":3.5270000000000001,"5min":4.0579999999999998,"15min":7.415},"last":0.221}]}]},{"osd":1,"up_from":8,"seq":34359739269,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5298116,"kb_used_data":3800252,"kb_used_omap":999,"kb_used_meta":1496856,"kb_avail":89073724,"statfs":{"total":96636764160,"available":91211493376,"internally_reserved":0,"allocated":3891458048,"data_stored":8741225955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1532781324},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,0,5,10,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.32500000000000001,"15min":0.374},"min":{"1min":0.19,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":0.49099999999999999,"5min":0.97899999999999998,"15min":5.7240000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.46600000000000003,"15min":0.48299999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.5,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.68700000000000006},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":2.7269999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.052,"5min":0.68100000000000005,"15min":0.62},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":6.1050000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.46800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5541112,"kb_used_data":3929200,"kb_used_omap":1020,"kb_used_meta":1610883,"kb_avail":88830728,"statfs":{"total":96636764160,"available":90962665472,"internally_reserved":0,"allocated":4023500800,"data_stored":9014073658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1649544860},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,4,3,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.68500000000000005,"15min":1.23},"min":{"1min":0.26400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.63800000000000001,"5min":17.991,"15min":207.047},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.86399999999999999,"15min":1.3440000000000001},"min":{"1min":0.24099999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.60999999999999999,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:47:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.77900000000000003,"15min":0.56899999999999995},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.67500000000000004,"5min":14.118,"15min":14.118},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.7000000000000002,"5min":14.051,"15min":14.051},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4021071872,"data_stored":9013047296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3888738304,"data_stored":8740311059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7905734656,"data_stored":17749282835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:48:20.517 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:20.517 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:20.518 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:20.518 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:20.518 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:48:23.241 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.227+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:48:23.241 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.231+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:48:23.259 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.251+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:48:23.260 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.251+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:48:23.260 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.251+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:48:23.260 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:48:23.251+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:48:40.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:48:41.088 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:48:41.089 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:48:41.124 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2411,"stamp":"2023-10-04T01:48:39.700906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27756806713,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2151777,"num_read_kb":7817097,"num_write":1994007,"num_write_kb":7809567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97742,"ondisk_log_size":97742,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20631988,"kb_used_data":15509148,"kb_used_omap":4402,"kb_used_meta":5118413,"kb_avail":262483532,"statfs":{"total":289910292480,"available":268783136768,"internally_reserved":0,"allocated":15881367552,"data_stored":35563954923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5241255262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,3,4,17,9,22],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18923520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4643,"num_read_kb":18504,"num_write":4624,"num_write_kb":18496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002012"},"pg_stats":[{"pgid":"2.7","version":"149'8239","reported_seq":11623,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.392583+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:48:36.392583+0000","last_peered":"2023-10-04T01:48:36.392583+0000","last_clean":"2023-10-04T01:48:36.392583+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:48:36.392583+0000","last_undegraded":"2023-10-04T01:48:36.392583+0000","last_fullsized":"2023-10-04T01:48:36.392583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8239,"log_dups_size":0,"ondisk_log_size":8239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'7371","reported_seq":11064,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:34.958922+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:48:34.958922+0000","last_peered":"2023-10-04T01:48:34.958922+0000","last_clean":"2023-10-04T01:48:34.958922+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:48:34.958922+0000","last_undegraded":"2023-10-04T01:48:34.958922+0000","last_fullsized":"2023-10-04T01:48:34.958922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7371,"log_dups_size":0,"ondisk_log_size":7371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'8382","reported_seq":12073,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.393510+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:48:36.393510+0000","last_peered":"2023-10-04T01:48:36.393510+0000","last_clean":"2023-10-04T01:48:36.393510+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:48:36.393510+0000","last_undegraded":"2023-10-04T01:48:36.393510+0000","last_fullsized":"2023-10-04T01:48:36.393510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'6812","reported_seq":11349,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.387664+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:48:36.387664+0000","last_peered":"2023-10-04T01:48:36.387664+0000","last_clean":"2023-10-04T01:48:36.387664+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:48:36.387664+0000","last_undegraded":"2023-10-04T01:48:36.387664+0000","last_fullsized":"2023-10-04T01:48:36.387664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'497345","reported_seq":979849,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:39.033522+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:48:39.033522+0000","last_peered":"2023-10-04T01:48:39.033522+0000","last_clean":"2023-10-04T01:48:39.033522+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:48:39.033522+0000","last_undegraded":"2023-10-04T01:48:39.033522+0000","last_fullsized":"2023-10-04T01:48:39.033522+0000","mapping_epoch":15,"log_start":"147'487273","ondisk_log_start":"147'487273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4041,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 574s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6971043840,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":487576,"num_read_kb":1919396,"num_write":487443,"num_write_kb":1949772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'17012","reported_seq":22187,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.804087+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:48:36.804087+0000","last_peered":"2023-10-04T01:48:36.804087+0000","last_clean":"2023-10-04T01:48:36.804087+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:48:36.804087+0000","last_undegraded":"2023-10-04T01:48:36.804087+0000","last_fullsized":"2023-10-04T01:48:36.804087+0000","mapping_epoch":12,"log_start":"81'7000","ondisk_log_start":"81'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'486283","reported_seq":958450,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:36.660524+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:48:36.660524+0000","last_peered":"2023-10-04T01:48:36.660524+0000","last_clean":"2023-10-04T01:48:36.660524+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:48:36.660524+0000","last_undegraded":"2023-10-04T01:48:36.660524+0000","last_fullsized":"2023-10-04T01:48:36.660524+0000","mapping_epoch":15,"log_start":"147'476266","ondisk_log_start":"147'476266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4197,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 569s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6824005632,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":477136,"num_read_kb":1878372,"num_write":476375,"num_write_kb":1905500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9050,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:34.060073+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:48:34.060073+0000","last_peered":"2023-10-04T01:48:34.060073+0000","last_clean":"2023-10-04T01:48:34.060073+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:48:34.060073+0000","last_undegraded":"2023-10-04T01:48:34.060073+0000","last_fullsized":"2023-10-04T01:48:34.060073+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'507677","reported_seq":1001397,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:39.003420+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:48:39.003420+0000","last_peered":"2023-10-04T01:48:39.003420+0000","last_clean":"2023-10-04T01:48:39.003420+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:48:39.003420+0000","last_undegraded":"2023-10-04T01:48:39.003420+0000","last_fullsized":"2023-10-04T01:48:39.003420+0000","mapping_epoch":15,"log_start":"147'497665","ondisk_log_start":"147'497665","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3648,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 570s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7140229120,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":498961,"num_read_kb":1965060,"num_write":497945,"num_write_kb":1991780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9588,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:34.059566+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:48:34.059566+0000","last_peered":"2023-10-04T01:48:34.059566+0000","last_clean":"2023-10-04T01:48:34.059566+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:48:34.059566+0000","last_undegraded":"2023-10-04T01:48:34.059566+0000","last_fullsized":"2023-10-04T01:48:34.059566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'483112","reported_seq":954916,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.677261+0000","last_change":"2023-10-04T01:48:36.296932+0000","last_active":"2023-10-04T01:48:36.677261+0000","last_peered":"2023-10-04T01:48:36.677261+0000","last_clean":"2023-10-04T01:48:36.677261+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:48:36.677261+0000","last_undegraded":"2023-10-04T01:48:36.677261+0000","last_fullsized":"2023-10-04T01:48:36.677261+0000","mapping_epoch":15,"log_start":"147'473063","ondisk_log_start":"147'473063","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6820921363,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":476930,"num_read_kb":1877416,"num_write":476600,"num_write_kb":1906394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'17946","reported_seq":23346,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:36.413593+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:48:36.413593+0000","last_peered":"2023-10-04T01:48:36.413593+0000","last_clean":"2023-10-04T01:48:36.413593+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:48:36.413593+0000","last_undegraded":"2023-10-04T01:48:36.413593+0000","last_fullsized":"2023-10-04T01:48:36.413593+0000","mapping_epoch":12,"log_start":"83'7900","ondisk_log_start":"83'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:41.388969+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:47:41.388969+0000","last_peered":"2023-10-04T01:47:41.388969+0000","last_clean":"2023-10-04T01:47:41.388969+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:47:41.388969+0000","last_undegraded":"2023-10-04T01:47:41.388969+0000","last_fullsized":"2023-10-04T01:47:41.388969+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27756199955,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1940603,"num_read_kb":7640244,"num_write":1938363,"num_write_kb":7753446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15875219456,"data_stored":35562315814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57493,"ondisk_log_size":57493,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739274,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9694080,"kb_used_data":7750584,"kb_used_omap":2383,"kb_used_meta":1941104,"kb_avail":84677760,"statfs":{"total":96636764160,"available":86710026240,"internally_reserved":0,"allocated":7936598016,"data_stored":17778860412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1987691446},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.48399999999999999,"15min":0.42399999999999999},"min":{"1min":0.20499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.702,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54600000000000004,"15min":0.48399999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":3.7440000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.27900000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.54700000000000004,"15min":0.46800000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4160000000000004,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.48399999999999999,"15min":0.45700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.19600000000000001},"max":{"1min":3.5270000000000001,"5min":4.0579999999999998,"15min":7.415},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739273,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5366588,"kb_used_data":3814580,"kb_used_omap":999,"kb_used_meta":1551000,"kb_avail":89005252,"statfs":{"total":96636764160,"available":91141378048,"internally_reserved":0,"allocated":3906129920,"data_stored":8755889932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1588224780},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.32500000000000001,"15min":0.374},"min":{"1min":0.19,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":0.49099999999999999,"5min":0.97899999999999998,"15min":5.7240000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.46600000000000003,"15min":0.48299999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.5,"5min":4.6429999999999998,"15min":5.6520000000000001},"last":0.23799999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.69799999999999995,"15min":0.68700000000000006},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":2.7269999999999999,"5min":10.494999999999999,"15min":14.467000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.68100000000000005,"15min":0.62},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":6.1050000000000004,"5min":10.531000000000001,"15min":14.503},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5571320,"kb_used_data":3943984,"kb_used_omap":1020,"kb_used_meta":1626307,"kb_avail":88800520,"statfs":{"total":96636764160,"available":90931732480,"internally_reserved":0,"allocated":4038639616,"data_stored":9029204579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1665339036},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,2,13,9,22],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:48:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.61899999999999999,"15min":1.232},"min":{"1min":0.214,"5min":0.188,"15min":0.188},"max":{"1min":0.59499999999999997,"5min":17.991,"15min":207.047},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.79700000000000004,"15min":1.3280000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.266,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:47:38 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.77900000000000003,"15min":0.56899999999999995},"min":{"1min":0.23799999999999999,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.67500000000000004,"5min":14.118,"15min":14.118},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.22600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.7000000000000002,"5min":14.051,"15min":14.051},"last":0.27600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4036149248,"data_stored":9028124672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3903389696,"data_stored":8754962451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7935680512,"data_stored":17779228691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:48:41.126 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:41.126 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:41.126 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:41.126 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:48:41.127 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:49:01.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:49:01.629 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:49:01.630 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:49:01.658 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2421,"stamp":"2023-10-04T01:48:59.709871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27786609209,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2159135,"num_read_kb":7846397,"num_write":2001296,"num_write_kb":7838723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97951,"ondisk_log_size":97951,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20861676,"kb_used_data":15567444,"kb_used_omap":4402,"kb_used_meta":5289805,"kb_avail":262253844,"statfs":{"total":289910292480,"available":268547936256,"internally_reserved":0,"allocated":15941062656,"data_stored":35623650027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5416760670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,1,6,15,14,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18608128,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4564,"num_read_kb":18184,"num_write":4547,"num_write_kb":18188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002098"},"pg_stats":[{"pgid":"2.7","version":"149'8273","reported_seq":11665,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:56.657364+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:48:56.657364+0000","last_peered":"2023-10-04T01:48:56.657364+0000","last_clean":"2023-10-04T01:48:56.657364+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:48:56.657364+0000","last_undegraded":"2023-10-04T01:48:56.657364+0000","last_fullsized":"2023-10-04T01:48:56.657364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'7388","reported_seq":11085,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:54.539256+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:48:54.539256+0000","last_peered":"2023-10-04T01:48:54.539256+0000","last_clean":"2023-10-04T01:48:54.539256+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:48:54.539256+0000","last_undegraded":"2023-10-04T01:48:54.539256+0000","last_fullsized":"2023-10-04T01:48:54.539256+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7388,"log_dups_size":0,"ondisk_log_size":7388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'8416","reported_seq":12116,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:56.652512+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:48:56.652512+0000","last_peered":"2023-10-04T01:48:56.652512+0000","last_clean":"2023-10-04T01:48:56.652512+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:48:56.652512+0000","last_undegraded":"2023-10-04T01:48:56.652512+0000","last_fullsized":"2023-10-04T01:48:56.652512+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'6846","reported_seq":11392,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:56.647044+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:48:56.647044+0000","last_peered":"2023-10-04T01:48:56.647044+0000","last_clean":"2023-10-04T01:48:56.647044+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:48:56.647044+0000","last_undegraded":"2023-10-04T01:48:56.647044+0000","last_fullsized":"2023-10-04T01:48:56.647044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6846,"log_dups_size":0,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'499244","reported_seq":983693,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:59.109035+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:48:59.109035+0000","last_peered":"2023-10-04T01:48:59.109035+0000","last_clean":"2023-10-04T01:48:59.109035+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:48:59.109035+0000","last_undegraded":"2023-10-04T01:48:59.109035+0000","last_fullsized":"2023-10-04T01:48:59.109035+0000","mapping_epoch":15,"log_start":"148'489173","ondisk_log_start":"148'489173","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4173,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 594s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6978351104,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":489511,"num_read_kb":1927112,"num_write":489235,"num_write_kb":1956940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'17036","reported_seq":22224,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:57.621709+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:48:57.621709+0000","last_peered":"2023-10-04T01:48:57.621709+0000","last_clean":"2023-10-04T01:48:57.621709+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:48:57.621709+0000","last_undegraded":"2023-10-04T01:48:57.621709+0000","last_fullsized":"2023-10-04T01:48:57.621709+0000","mapping_epoch":12,"log_start":"81'7000","ondisk_log_start":"81'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'488185","reported_seq":962117,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:56.771233+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:48:56.771233+0000","last_peered":"2023-10-04T01:48:56.771233+0000","last_clean":"2023-10-04T01:48:56.771233+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:48:56.771233+0000","last_undegraded":"2023-10-04T01:48:56.771233+0000","last_fullsized":"2023-10-04T01:48:56.771233+0000","mapping_epoch":15,"log_start":"147'478166","ondisk_log_start":"147'478166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4326,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 589s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6831374336,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":478888,"num_read_kb":1885352,"num_write":478171,"num_write_kb":1912684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9066,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:54.542811+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:48:54.542811+0000","last_peered":"2023-10-04T01:48:54.542811+0000","last_clean":"2023-10-04T01:48:54.542811+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:48:54.542811+0000","last_undegraded":"2023-10-04T01:48:54.542811+0000","last_fullsized":"2023-10-04T01:48:54.542811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'509680","reported_seq":1005257,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:48:59.116301+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:48:59.116301+0000","last_peered":"2023-10-04T01:48:59.116301+0000","last_clean":"2023-10-04T01:48:59.116301+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:48:59.116301+0000","last_undegraded":"2023-10-04T01:48:59.116301+0000","last_fullsized":"2023-10-04T01:48:59.116301+0000","mapping_epoch":15,"log_start":"148'499666","ondisk_log_start":"148'499666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3791,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 590s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7147941888,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":500804,"num_read_kb":1972388,"num_write":499834,"num_write_kb":1999336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9600,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:54.539939+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:48:54.539939+0000","last_peered":"2023-10-04T01:48:54.539939+0000","last_clean":"2023-10-04T01:48:54.539939+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:48:54.539939+0000","last_undegraded":"2023-10-04T01:48:54.539939+0000","last_fullsized":"2023-10-04T01:48:54.539939+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'484924","reported_seq":958563,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:56.748158+0000","last_change":"2023-10-04T01:48:54.438367+0000","last_active":"2023-10-04T01:48:56.748158+0000","last_peered":"2023-10-04T01:48:56.748158+0000","last_clean":"2023-10-04T01:48:56.748158+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:48:56.748158+0000","last_undegraded":"2023-10-04T01:48:56.748158+0000","last_fullsized":"2023-10-04T01:48:56.748158+0000","mapping_epoch":15,"log_start":"147'474863","ondisk_log_start":"147'474863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6828335123,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":478758,"num_read_kb":1884692,"num_write":478412,"num_write_kb":1913642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'17997","reported_seq":23410,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:56.688154+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:48:56.688154+0000","last_peered":"2023-10-04T01:48:56.688154+0000","last_clean":"2023-10-04T01:48:56.688154+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:48:56.688154+0000","last_undegraded":"2023-10-04T01:48:56.688154+0000","last_fullsized":"2023-10-04T01:48:56.688154+0000","mapping_epoch":12,"log_start":"83'7900","ondisk_log_start":"83'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:41.388969+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:47:41.388969+0000","last_peered":"2023-10-04T01:47:41.388969+0000","last_clean":"2023-10-04T01:47:41.388969+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:47:41.388969+0000","last_undegraded":"2023-10-04T01:47:41.388969+0000","last_fullsized":"2023-10-04T01:47:41.388969+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27786002451,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1947961,"num_read_kb":7669544,"num_write":1945652,"num_write_kb":7782602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15935086592,"data_stored":35622182950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57687,"ondisk_log_size":57687,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739278,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9791160,"kb_used_data":7779632,"kb_used_omap":2383,"kb_used_meta":2009136,"kb_avail":84580680,"statfs":{"total":96636764160,"available":86610616320,"internally_reserved":0,"allocated":7966343168,"data_stored":17808605564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":2057356214},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.16500000000000001},"max":{"1min":4.3780000000000001,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.61299999999999999,"15min":0.50900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":4.4580000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4169999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.48999999999999999,"15min":0.46999999999999997},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":4.5419999999999998,"5min":4.5419999999999998,"15min":7.415},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739277,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5423696,"kb_used_data":3829128,"kb_used_omap":999,"kb_used_meta":1593560,"kb_avail":88948144,"statfs":{"total":96636764160,"available":91082899456,"internally_reserved":0,"allocated":3921027072,"data_stored":8770787084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1631806220},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.34699999999999998,"15min":0.375},"min":{"1min":0.23200000000000001,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.55100000000000005,"15min":0.50800000000000001},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5730000000000004,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.50700000000000001,"15min":0.70599999999999996},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":14.467000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.58599999999999997,"15min":0.64000000000000001},"min":{"1min":0.25700000000000001,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":3.601,"5min":6.1050000000000004,"15min":14.503},"last":0.42099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739278,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5646820,"kb_used_data":3958684,"kb_used_omap":1020,"kb_used_meta":1687107,"kb_avail":88725020,"statfs":{"total":96636764160,"available":90854420480,"internally_reserved":0,"allocated":4053692416,"data_stored":9044257379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1727598236},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,10,6,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:48:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.61899999999999999,"15min":1.232},"min":{"1min":0.214,"5min":0.188,"15min":0.188},"max":{"1min":0.59499999999999997,"5min":17.991,"15min":207.047},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.79700000000000004,"15min":1.3280000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.266,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:48:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.67000000000000004,"15min":0.56999999999999995},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.501,"5min":14.118,"15min":14.118},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60099999999999998,"15min":0.52800000000000002},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0720000000000001,"5min":14.051,"15min":14.051},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4051243008,"data_stored":9043218432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3918163968,"data_stored":8769736723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7965679616,"data_stored":17809227795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:49:01.659 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:01.659 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:01.660 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:01.660 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:01.660 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:49:17.478 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.458+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:49:17.478 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:49:17.478 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:49:17.479 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:49:17.479 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:49:17.479 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:49:17.479 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:49:17.479 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:49:17.480 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:17.466+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:49:18.408 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:18.398+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:49:18.409 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:49:18.398+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:49:21.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:49:22.167 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:49:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:49:22.215 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2432,"stamp":"2023-10-04T01:49:21.713810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27817140793,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2166484,"num_read_kb":7875641,"num_write":2008758,"num_write_kb":7868571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98072,"ondisk_log_size":98072,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21200312,"kb_used_data":15633820,"kb_used_omap":4402,"kb_used_meta":5562061,"kb_avail":261915208,"statfs":{"total":289910292480,"available":268201172992,"internally_reserved":0,"allocated":16009031680,"data_stored":35691619051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5695550814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,6,8,3,21,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15409152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3712,"num_read_kb":14764,"num_write":3775,"num_write_kb":15100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002077"},"pg_stats":[{"pgid":"2.7","version":"149'8297","reported_seq":11697,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:15.022832+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:49:15.022832+0000","last_peered":"2023-10-04T01:49:15.022832+0000","last_clean":"2023-10-04T01:49:15.022832+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:49:15.022832+0000","last_undegraded":"2023-10-04T01:49:15.022832+0000","last_fullsized":"2023-10-04T01:49:15.022832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8297,"log_dups_size":0,"ondisk_log_size":8297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'7400","reported_seq":11101,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:14.077471+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:49:14.077471+0000","last_peered":"2023-10-04T01:49:14.077471+0000","last_clean":"2023-10-04T01:49:14.077471+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:49:14.077471+0000","last_undegraded":"2023-10-04T01:49:14.077471+0000","last_fullsized":"2023-10-04T01:49:14.077471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'8440","reported_seq":12148,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:15.022614+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:49:15.022614+0000","last_peered":"2023-10-04T01:49:15.022614+0000","last_clean":"2023-10-04T01:49:15.022614+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:49:15.022614+0000","last_undegraded":"2023-10-04T01:49:15.022614+0000","last_fullsized":"2023-10-04T01:49:15.022614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'6870","reported_seq":11424,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:15.022633+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:49:15.022633+0000","last_peered":"2023-10-04T01:49:15.022633+0000","last_clean":"2023-10-04T01:49:15.022633+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:49:15.022633+0000","last_undegraded":"2023-10-04T01:49:15.022633+0000","last_fullsized":"2023-10-04T01:49:15.022633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'501273","reported_seq":987595,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:19.433573+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:49:19.433573+0000","last_peered":"2023-10-04T01:49:19.433573+0000","last_clean":"2023-10-04T01:49:19.433573+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:49:19.433573+0000","last_undegraded":"2023-10-04T01:49:19.433573+0000","last_fullsized":"2023-10-04T01:49:19.433573+0000","mapping_epoch":15,"log_start":"148'491173","ondisk_log_start":"148'491173","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4302,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 615s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6986223616,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":491374,"num_read_kb":1934532,"num_write":491160,"num_write_kb":1964640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"149'17060","reported_seq":22260,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:19.071382+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:49:19.071382+0000","last_peered":"2023-10-04T01:49:19.071382+0000","last_clean":"2023-10-04T01:49:19.071382+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:49:19.071382+0000","last_undegraded":"2023-10-04T01:49:19.071382+0000","last_fullsized":"2023-10-04T01:49:19.071382+0000","mapping_epoch":12,"log_start":"81'7000","ondisk_log_start":"81'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"149'490206","reported_seq":965957,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:16.940337+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:49:16.940337+0000","last_peered":"2023-10-04T01:49:16.940337+0000","last_clean":"2023-10-04T01:49:16.940337+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:49:16.940337+0000","last_undegraded":"2023-10-04T01:49:16.940337+0000","last_fullsized":"2023-10-04T01:49:16.940337+0000","mapping_epoch":15,"log_start":"148'480166","ondisk_log_start":"148'480166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4460,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 609s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6839189504,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":480707,"num_read_kb":1892584,"num_write":480080,"num_write_kb":1920320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9086,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:20.144749+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:49:20.144749+0000","last_peered":"2023-10-04T01:49:20.144749+0000","last_clean":"2023-10-04T01:49:20.144749+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:49:20.144749+0000","last_undegraded":"2023-10-04T01:49:20.144749+0000","last_fullsized":"2023-10-04T01:49:20.144749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"149'511666","reported_seq":1009183,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:19.426343+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:49:19.426343+0000","last_peered":"2023-10-04T01:49:19.426343+0000","last_clean":"2023-10-04T01:49:19.426343+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:49:19.426343+0000","last_undegraded":"2023-10-04T01:49:19.426343+0000","last_fullsized":"2023-10-04T01:49:19.426343+0000","mapping_epoch":15,"log_start":"148'501566","ondisk_log_start":"148'501566","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3916,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7155630080,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":502732,"num_read_kb":1980052,"num_write":501721,"num_write_kb":2006884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9615,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:20.142895+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:49:20.142895+0000","last_peered":"2023-10-04T01:49:20.142895+0000","last_clean":"2023-10-04T01:49:20.142895+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:49:20.142895+0000","last_undegraded":"2023-10-04T01:49:20.142895+0000","last_fullsized":"2023-10-04T01:49:20.142895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"149'486665","reported_seq":962047,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:16.965934+0000","last_change":"2023-10-04T01:49:11.466922+0000","last_active":"2023-10-04T01:49:16.965934+0000","last_peered":"2023-10-04T01:49:16.965934+0000","last_clean":"2023-10-04T01:49:16.965934+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:49:16.965934+0000","last_undegraded":"2023-10-04T01:49:16.965934+0000","last_fullsized":"2023-10-04T01:49:16.965934+0000","mapping_epoch":15,"log_start":"148'476663","ondisk_log_start":"148'476663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6835490835,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":480497,"num_read_kb":1891620,"num_write":480153,"num_write_kb":1920606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"149'18033","reported_seq":23458,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:49:15.022848+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:49:15.022848+0000","last_peered":"2023-10-04T01:49:15.022848+0000","last_clean":"2023-10-04T01:49:15.022848+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:49:15.022848+0000","last_undegraded":"2023-10-04T01:49:15.022848+0000","last_fullsized":"2023-10-04T01:49:15.022848+0000","mapping_epoch":12,"log_start":"83'8000","ondisk_log_start":"83'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-04T01:47:41.388969+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:47:41.388969+0000","last_peered":"2023-10-04T01:47:41.388969+0000","last_clean":"2023-10-04T01:47:41.388969+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:47:41.388969+0000","last_undegraded":"2023-10-04T01:47:41.388969+0000","last_fullsized":"2023-10-04T01:47:41.388969+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27816534035,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1955310,"num_read_kb":7698788,"num_write":1953114,"num_write_kb":7812450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16003211264,"data_stored":35690307622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40242,"ondisk_log_size":40242,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57731,"ondisk_log_size":57731,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9968684,"kb_used_data":7816676,"kb_used_omap":2383,"kb_used_meta":2149616,"kb_avail":84403156,"statfs":{"total":96636764160,"available":86428831744,"internally_reserved":0,"allocated":8004276224,"data_stored":17846538620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":2201207734},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.16500000000000001},"max":{"1min":4.3780000000000001,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.61299999999999999,"15min":0.50900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":4.4580000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":1.03}]},{"osd":1,"last update":"Wed Oct 4 01:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4169999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":2.6949999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.48999999999999999,"15min":0.46999999999999997},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":4.5419999999999998,"5min":4.5419999999999998,"15min":7.415},"last":0.51200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5508896,"kb_used_data":3843544,"kb_used_omap":999,"kb_used_meta":1664344,"kb_avail":88862944,"statfs":{"total":96636764160,"available":90995654656,"internally_reserved":0,"allocated":3935789056,"data_stored":8785549068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1704289036},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.34699999999999998,"15min":0.375},"min":{"1min":0.23200000000000001,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.214},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.55100000000000005,"15min":0.50800000000000001},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5730000000000004,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.50700000000000001,"15min":0.70599999999999996},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":14.467000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.58599999999999997,"15min":0.64000000000000001},"min":{"1min":0.25700000000000001,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":3.601,"5min":6.1050000000000004,"15min":14.503},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5722732,"kb_used_data":3973600,"kb_used_omap":1020,"kb_used_meta":1748099,"kb_avail":88649108,"statfs":{"total":96636764160,"available":90776686592,"internally_reserved":0,"allocated":4068966400,"data_stored":9059531363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1790054044},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,1,18,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:48:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.61899999999999999,"15min":1.232},"min":{"1min":0.214,"5min":0.188,"15min":0.188},"max":{"1min":0.59499999999999997,"5min":17.991,"15min":207.047},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.79700000000000004,"15min":1.3280000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.266,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.35299999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:48:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.67000000000000004,"15min":0.56999999999999995},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.501,"5min":14.118,"15min":14.118},"last":0.247},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60099999999999998,"15min":0.52800000000000002},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0720000000000001,"5min":14.051,"15min":14.051},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4066848768,"data_stored":9058824192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3933118464,"data_stored":8784691219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8003244032,"data_stored":17846792211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:49:22.217 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:22.217 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:22.218 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:22.218 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:22.218 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:49:29.528 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:29.525+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:49:29.528 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:29.525+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:49:29.528 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:29.525+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:49:36.636 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:36.629+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:49:42.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:49:42.735 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:49:42.735 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:49:42.770 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2443,"stamp":"2023-10-04T01:49:41.717344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27847340601,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2173515,"num_read_kb":7903625,"num_write":2016143,"num_write_kb":7898111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98153,"ondisk_log_size":98153,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21467624,"kb_used_data":15692752,"kb_used_omap":4402,"kb_used_meta":5770445,"kb_avail":261647896,"statfs":{"total":289910292480,"available":267927445504,"internally_reserved":0,"allocated":16069378048,"data_stored":35751953893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5908936030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,14,10,14,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15527936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3588,"num_read_kb":14284,"num_write":3790,"num_write_kb":15160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001730"},"pg_stats":[{"pgid":"2.7","version":"150'8309","reported_seq":11719,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.499213+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:49:35.499213+0000","last_peered":"2023-10-04T01:49:35.499213+0000","last_clean":"2023-10-04T01:49:35.499213+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:49:35.499213+0000","last_undegraded":"2023-10-04T01:49:35.499213+0000","last_fullsized":"2023-10-04T01:49:35.499213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'7407","reported_seq":11114,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:34.650390+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:49:34.650390+0000","last_peered":"2023-10-04T01:49:34.650390+0000","last_clean":"2023-10-04T01:49:34.650390+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:49:34.650390+0000","last_undegraded":"2023-10-04T01:49:34.650390+0000","last_fullsized":"2023-10-04T01:49:34.650390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'8453","reported_seq":12170,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.498648+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:49:35.498648+0000","last_peered":"2023-10-04T01:49:35.498648+0000","last_clean":"2023-10-04T01:49:35.498648+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:49:35.498648+0000","last_undegraded":"2023-10-04T01:49:35.498648+0000","last_fullsized":"2023-10-04T01:49:35.498648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'6882","reported_seq":11445,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.498667+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:49:35.498667+0000","last_peered":"2023-10-04T01:49:35.498667+0000","last_clean":"2023-10-04T01:49:35.498667+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:49:35.498667+0000","last_undegraded":"2023-10-04T01:49:35.498667+0000","last_fullsized":"2023-10-04T01:49:35.498667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'503160","reported_seq":991278,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:39.558387+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:49:39.558387+0000","last_peered":"2023-10-04T01:49:39.558387+0000","last_clean":"2023-10-04T01:49:39.558387+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:49:39.558387+0000","last_undegraded":"2023-10-04T01:49:39.558387+0000","last_fullsized":"2023-10-04T01:49:39.558387+0000","mapping_epoch":15,"log_start":"148'493073","ondisk_log_start":"148'493073","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4427,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 635s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":6993539072,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":493163,"num_read_kb":1941652,"num_write":492948,"num_write_kb":1971792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'17087","reported_seq":22300,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:36.674698+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:49:36.674698+0000","last_peered":"2023-10-04T01:49:36.674698+0000","last_clean":"2023-10-04T01:49:36.674698+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:49:36.674698+0000","last_undegraded":"2023-10-04T01:49:36.674698+0000","last_fullsized":"2023-10-04T01:49:36.674698+0000","mapping_epoch":12,"log_start":"81'7000","ondisk_log_start":"81'7000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'492117","reported_seq":969595,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:37.078307+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:49:37.078307+0000","last_peered":"2023-10-04T01:49:37.078307+0000","last_clean":"2023-10-04T01:49:37.078307+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:49:37.078307+0000","last_undegraded":"2023-10-04T01:49:37.078307+0000","last_fullsized":"2023-10-04T01:49:37.078307+0000","mapping_epoch":15,"log_start":"148'482066","ondisk_log_start":"148'482066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4598,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 629s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6846554112,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":482417,"num_read_kb":1899400,"num_write":481880,"num_write_kb":1927520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9099,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.506532+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:49:35.506532+0000","last_peered":"2023-10-04T01:49:35.506532+0000","last_clean":"2023-10-04T01:49:35.506532+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:49:35.506532+0000","last_undegraded":"2023-10-04T01:49:35.506532+0000","last_fullsized":"2023-10-04T01:49:35.506532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'513706","reported_seq":1012992,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:39.561296+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:49:39.561296+0000","last_peered":"2023-10-04T01:49:39.561296+0000","last_clean":"2023-10-04T01:49:39.561296+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:49:39.561296+0000","last_undegraded":"2023-10-04T01:49:39.561296+0000","last_fullsized":"2023-10-04T01:49:39.561296+0000","mapping_epoch":15,"log_start":"148'503666","ondisk_log_start":"148'503666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4037,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 631s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7163564032,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":504497,"num_read_kb":1987060,"num_write":503665,"num_write_kb":2014660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9625,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.500632+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:49:35.500632+0000","last_peered":"2023-10-04T01:49:35.500632+0000","last_clean":"2023-10-04T01:49:35.500632+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:49:35.500632+0000","last_undegraded":"2023-10-04T01:49:35.500632+0000","last_fullsized":"2023-10-04T01:49:35.500632+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'488518","reported_seq":965671,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:37.070501+0000","last_change":"2023-10-04T01:49:36.497465+0000","last_active":"2023-10-04T01:49:37.070501+0000","last_peered":"2023-10-04T01:49:37.070501+0000","last_clean":"2023-10-04T01:49:37.070501+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:49:37.070501+0000","last_undegraded":"2023-10-04T01:49:37.070501+0000","last_fullsized":"2023-10-04T01:49:37.070501+0000","mapping_epoch":15,"log_start":"148'478463","ondisk_log_start":"148'478463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6843076627,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":482264,"num_read_kb":1898660,"num_write":482006,"num_write_kb":1928018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'18052","reported_seq":23490,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:35.499346+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:49:35.499346+0000","last_peered":"2023-10-04T01:49:35.499346+0000","last_clean":"2023-10-04T01:49:35.499346+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:49:35.499346+0000","last_undegraded":"2023-10-04T01:49:35.499346+0000","last_fullsized":"2023-10-04T01:49:35.499346+0000","mapping_epoch":12,"log_start":"83'8000","ondisk_log_start":"83'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":325,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-04T01:48:23.296559+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:48:23.296559+0000","last_peered":"2023-10-04T01:48:23.296559+0000","last_clean":"2023-10-04T01:48:23.296559+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:48:23.296559+0000","last_undegraded":"2023-10-04T01:48:23.296559+0000","last_fullsized":"2023-10-04T01:48:23.296559+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27846733843,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1962341,"num_read_kb":7726772,"num_write":1960499,"num_write_kb":7841990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16063815680,"data_stored":35750912038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40233,"ondisk_log_size":40233,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57821,"ondisk_log_size":57821,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10138592,"kb_used_data":7846232,"kb_used_omap":2383,"kb_used_meta":2289968,"kb_avail":84233248,"statfs":{"total":96636764160,"available":86254845952,"internally_reserved":0,"allocated":8034541568,"data_stored":17876800122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":2344928182},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:52 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.16500000000000001},"max":{"1min":4.3780000000000001,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.61299999999999999,"15min":0.50900000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22900000000000001},"max":{"1min":4.4580000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":4.4169999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.31},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.48999999999999999,"15min":0.46999999999999997},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":4.5419999999999998,"5min":4.5419999999999998,"15min":7.415},"last":0.51000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5555284,"kb_used_data":3857996,"kb_used_omap":999,"kb_used_meta":1696280,"kb_avail":88816556,"statfs":{"total":96636764160,"available":90948153344,"internally_reserved":0,"allocated":3950587904,"data_stored":8800344074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1736991500},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.34699999999999998,"15min":0.375},"min":{"1min":0.23200000000000001,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.55100000000000005,"15min":0.50800000000000001},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5730000000000004,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Oct 4 01:48:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.50700000000000001,"15min":0.70599999999999996},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.17999999999999999},"max":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":14.467000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.58599999999999997,"15min":0.64000000000000001},"min":{"1min":0.25700000000000001,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":3.601,"5min":6.1050000000000004,"15min":14.503},"last":1.466}]}]},{"osd":0,"up_from":8,"seq":34359739286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5773748,"kb_used_data":3988524,"kb_used_omap":1020,"kb_used_meta":1784195,"kb_avail":88598092,"statfs":{"total":96636764160,"available":90724446208,"internally_reserved":0,"allocated":4084248576,"data_stored":9074809697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1827016348},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,3,9,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.61299999999999999,"15min":1.234},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.46999999999999997,"5min":17.991,"15min":207.047},"last":0.252},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.79100000000000004,"15min":1.323},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80200000000000005,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.32700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:48:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.67000000000000004,"15min":0.56999999999999995},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.501,"5min":14.118,"15min":14.118},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60099999999999998,"15min":0.52800000000000002},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0720000000000001,"5min":14.051,"15min":14.051},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4082126848,"data_stored":9074102272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3948077056,"data_stored":8799649811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8033611776,"data_stored":17877159955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:49:42.772 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:42.772 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:42.773 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:42.773 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:49:42.773 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:49:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:49:43.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:49:43.844 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:49:43.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:49:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:49:44.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:49:44.725 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:44.721+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:49:44.725 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:44.721+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T01:49:44.725 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:44.721+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:49:44.725 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:49:44.721+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:49:45.019 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:49:45.055 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:50:05.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:50:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:50:05.535 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:50:05.567 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2454,"stamp":"2023-10-04T01:50:03.721164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27879973433,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2181321,"num_read_kb":7934745,"num_write":2024123,"num_write_kb":7930031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98317,"ondisk_log_size":98317,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20999020,"kb_used_data":15752916,"kb_used_omap":4402,"kb_used_meta":5241677,"kb_avail":262116500,"statfs":{"total":289910292480,"available":268407296000,"internally_reserved":0,"allocated":16130985984,"data_stored":35813561829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5367477598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,4,11,20,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18100224,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4204,"num_read_kb":16764,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002094"},"pg_stats":[{"pgid":"2.7","version":"150'8341","reported_seq":11761,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:01.099217+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:50:01.099217+0000","last_peered":"2023-10-04T01:50:01.099217+0000","last_clean":"2023-10-04T01:50:01.099217+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:50:01.099217+0000","last_undegraded":"2023-10-04T01:50:01.099217+0000","last_fullsized":"2023-10-04T01:50:01.099217+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8341,"log_dups_size":0,"ondisk_log_size":8341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'7423","reported_seq":11135,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:00.612528+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:50:00.612528+0000","last_peered":"2023-10-04T01:50:00.612528+0000","last_clean":"2023-10-04T01:50:00.612528+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:50:00.612528+0000","last_undegraded":"2023-10-04T01:50:00.612528+0000","last_fullsized":"2023-10-04T01:50:00.612528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'8485","reported_seq":12212,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:01.098570+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:50:01.098570+0000","last_peered":"2023-10-04T01:50:01.098570+0000","last_clean":"2023-10-04T01:50:01.098570+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:50:01.098570+0000","last_undegraded":"2023-10-04T01:50:01.098570+0000","last_fullsized":"2023-10-04T01:50:01.098570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'6914","reported_seq":11487,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:01.098875+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:50:01.098875+0000","last_peered":"2023-10-04T01:50:01.098875+0000","last_clean":"2023-10-04T01:50:01.098875+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:50:01.098875+0000","last_undegraded":"2023-10-04T01:50:01.098875+0000","last_fullsized":"2023-10-04T01:50:01.098875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'505031","reported_seq":994962,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:59.786932+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:49:59.786932+0000","last_peered":"2023-10-04T01:49:59.786932+0000","last_clean":"2023-10-04T01:49:59.786932+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:49:59.786932+0000","last_undegraded":"2023-10-04T01:49:59.786932+0000","last_fullsized":"2023-10-04T01:49:59.786932+0000","mapping_epoch":15,"log_start":"148'494973","ondisk_log_start":"148'494973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4576,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 655s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7000707072,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":494957,"num_read_kb":1948780,"num_write":494697,"num_write_kb":1978788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'17136","reported_seq":22361,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:58.412289+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:49:58.412289+0000","last_peered":"2023-10-04T01:49:58.412289+0000","last_clean":"2023-10-04T01:49:58.412289+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:49:58.412289+0000","last_undegraded":"2023-10-04T01:49:58.412289+0000","last_fullsized":"2023-10-04T01:49:58.412289+0000","mapping_epoch":12,"log_start":"81'7100","ondisk_log_start":"81'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'494431","reported_seq":974047,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:02.165791+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:50:02.165791+0000","last_peered":"2023-10-04T01:50:02.165791+0000","last_clean":"2023-10-04T01:50:02.165791+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:50:02.165791+0000","last_undegraded":"2023-10-04T01:50:02.165791+0000","last_fullsized":"2023-10-04T01:50:02.165791+0000","mapping_epoch":15,"log_start":"148'484367","ondisk_log_start":"148'484367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4780,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 654s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6855389184,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":484536,"num_read_kb":1907848,"num_write":484046,"num_write_kb":1936184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9115,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:56.058800+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:49:56.058800+0000","last_peered":"2023-10-04T01:49:56.058800+0000","last_clean":"2023-10-04T01:49:56.058800+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:49:56.058800+0000","last_undegraded":"2023-10-04T01:49:56.058800+0000","last_fullsized":"2023-10-04T01:49:56.058800+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'515654","reported_seq":1016719,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:49:59.768296+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:49:59.768296+0000","last_peered":"2023-10-04T01:49:59.768296+0000","last_clean":"2023-10-04T01:49:59.768296+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:49:59.768296+0000","last_undegraded":"2023-10-04T01:49:59.768296+0000","last_fullsized":"2023-10-04T01:49:59.768296+0000","mapping_epoch":15,"log_start":"148'505566","ondisk_log_start":"148'505566","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4163,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 651s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7171084288,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":506261,"num_read_kb":1994096,"num_write":505507,"num_write_kb":2022028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9638,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:55.981200+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:49:55.981200+0000","last_peered":"2023-10-04T01:49:55.981200+0000","last_clean":"2023-10-04T01:49:55.981200+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:49:55.981200+0000","last_undegraded":"2023-10-04T01:49:55.981200+0000","last_fullsized":"2023-10-04T01:49:55.981200+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'490741","reported_seq":970032,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:02.166097+0000","last_change":"2023-10-04T01:49:59.531655+0000","last_active":"2023-10-04T01:50:02.166097+0000","last_peered":"2023-10-04T01:50:02.166097+0000","last_clean":"2023-10-04T01:50:02.166097+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:50:02.166097+0000","last_undegraded":"2023-10-04T01:50:02.166097+0000","last_fullsized":"2023-10-04T01:50:02.166097+0000","mapping_epoch":15,"log_start":"148'480663","ondisk_log_start":"148'480663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6852186131,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":484393,"num_read_kb":1907168,"num_write":484229,"num_write_kb":1936910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'18100","reported_seq":23553,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:01.099440+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:50:01.099440+0000","last_peered":"2023-10-04T01:50:01.099440+0000","last_clean":"2023-10-04T01:50:01.099440+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:50:01.099440+0000","last_undegraded":"2023-10-04T01:50:01.099440+0000","last_fullsized":"2023-10-04T01:50:01.099440+0000","mapping_epoch":12,"log_start":"83'8000","ondisk_log_start":"83'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":326,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:33.318906+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:49:33.318906+0000","last_peered":"2023-10-04T01:49:33.318906+0000","last_clean":"2023-10-04T01:49:33.318906+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:49:33.318906+0000","last_undegraded":"2023-10-04T01:49:33.318906+0000","last_fullsized":"2023-10-04T01:49:33.318906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27879366675,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1970147,"num_read_kb":7757892,"num_write":1968479,"num_write_kb":7873910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16125259776,"data_stored":35812356134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40288,"ondisk_log_size":40288,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57930,"ondisk_log_size":57930,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739292,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9551236,"kb_used_data":7874300,"kb_used_omap":2383,"kb_used_meta":1674544,"kb_avail":84820604,"statfs":{"total":96636764160,"available":86856298496,"internally_reserved":0,"allocated":8063283200,"data_stored":17905541754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1714734006},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":1.1539999999999999,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.61399999999999999,"15min":0.51100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":1.1910000000000001,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:49:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.624,"15min":0.46800000000000003},"min":{"1min":0.23699999999999999,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":2.6949999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.48299999999999998,"15min":0.47399999999999998},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.68200000000000005,"5min":4.5419999999999998,"15min":7.415},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739290,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5605448,"kb_used_data":3875648,"kb_used_omap":999,"kb_used_meta":1728792,"kb_avail":88766392,"statfs":{"total":96636764160,"available":90896785408,"internally_reserved":0,"allocated":3968663552,"data_stored":8818419722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1770283788},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,1,7,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.33900000000000002,"15min":0.375},"min":{"1min":0.185,"5min":0.185,"15min":0.17899999999999999},"max":{"1min":0.39400000000000002,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.861,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.49099999999999999,"15min":0.65700000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":0.56100000000000005,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.58799999999999997,"15min":0.59799999999999998},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":1.466,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5842336,"kb_used_data":4002968,"kb_used_omap":1020,"kb_used_meta":1838339,"kb_avail":88529504,"statfs":{"total":96636764160,"available":90654212096,"internally_reserved":0,"allocated":4099039232,"data_stored":9089600353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1882459804},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,12,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.61299999999999999,"15min":1.234},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.46999999999999997,"5min":17.991,"15min":207.047},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.79100000000000004,"15min":1.323},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80200000000000005,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:49:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.61299999999999999,"15min":0.56999999999999995},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.46300000000000002,"5min":14.118,"15min":14.118},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.61099999999999999,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55400000000000005,"5min":14.051,"15min":14.051},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096835584,"data_stored":9088811008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3966058496,"data_stored":8817631251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8062365696,"data_stored":17905913875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:50:05.568 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:05.569 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:05.569 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:05.569 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:05.569 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:50:18.810 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:50:18.804+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:50:18.811 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:50:18.804+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:50:25.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:50:26.191 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:50:26.192 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:50:26.230 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2465,"stamp":"2023-10-04T01:50:25.724926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27913724473,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2189319,"num_read_kb":7966625,"num_write":2032369,"num_write_kb":7963015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98359,"ondisk_log_size":98359,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21259056,"kb_used_data":15815892,"kb_used_omap":4402,"kb_used_meta":5438733,"kb_avail":261856464,"statfs":{"total":289910292480,"available":268141019136,"internally_reserved":0,"allocated":16195473408,"data_stored":35878049253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5569262942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,4,3,6,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19054592,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4604,"num_read_kb":18340,"num_write":4651,"num_write_kb":18604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002041"},"pg_stats":[{"pgid":"2.7","version":"150'8363","reported_seq":11791,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:21.869898+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:50:21.869898+0000","last_peered":"2023-10-04T01:50:21.869898+0000","last_clean":"2023-10-04T01:50:21.869898+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:50:21.869898+0000","last_undegraded":"2023-10-04T01:50:21.869898+0000","last_fullsized":"2023-10-04T01:50:21.869898+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'7434","reported_seq":11150,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:21.579018+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:50:21.579018+0000","last_peered":"2023-10-04T01:50:21.579018+0000","last_clean":"2023-10-04T01:50:21.579018+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:50:21.579018+0000","last_undegraded":"2023-10-04T01:50:21.579018+0000","last_fullsized":"2023-10-04T01:50:21.579018+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7434,"log_dups_size":0,"ondisk_log_size":7434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'8507","reported_seq":12242,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:21.865653+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:50:21.865653+0000","last_peered":"2023-10-04T01:50:21.865653+0000","last_clean":"2023-10-04T01:50:21.865653+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:50:21.865653+0000","last_undegraded":"2023-10-04T01:50:21.865653+0000","last_fullsized":"2023-10-04T01:50:21.865653+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'6936","reported_seq":11517,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:21.869838+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:50:21.869838+0000","last_peered":"2023-10-04T01:50:21.869838+0000","last_clean":"2023-10-04T01:50:21.869838+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:50:21.869838+0000","last_undegraded":"2023-10-04T01:50:21.869838+0000","last_fullsized":"2023-10-04T01:50:21.869838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'507463","reported_seq":999646,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:24.978504+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:50:24.978504+0000","last_peered":"2023-10-04T01:50:24.978504+0000","last_clean":"2023-10-04T01:50:24.978504+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:50:24.978504+0000","last_undegraded":"2023-10-04T01:50:24.978504+0000","last_fullsized":"2023-10-04T01:50:24.978504+0000","mapping_epoch":15,"log_start":"149'497373","ondisk_log_start":"149'497373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4794,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 680s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7009927168,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":497195,"num_read_kb":1957708,"num_write":496952,"num_write_kb":1987808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'17187","reported_seq":22427,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:23.228672+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:50:23.228672+0000","last_peered":"2023-10-04T01:50:23.228672+0000","last_clean":"2023-10-04T01:50:23.228672+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:50:23.228672+0000","last_undegraded":"2023-10-04T01:50:23.228672+0000","last_fullsized":"2023-10-04T01:50:23.228672+0000","mapping_epoch":12,"log_start":"81'7100","ondisk_log_start":"81'7100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'496411","reported_seq":977789,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:22.216490+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:50:22.216490+0000","last_peered":"2023-10-04T01:50:22.216490+0000","last_clean":"2023-10-04T01:50:22.216490+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:50:22.216490+0000","last_undegraded":"2023-10-04T01:50:22.216490+0000","last_fullsized":"2023-10-04T01:50:22.216490+0000","mapping_epoch":15,"log_start":"149'486367","ondisk_log_start":"149'486367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4930,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 674s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6862958592,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":486287,"num_read_kb":1914824,"num_write":485900,"num_write_kb":1943600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9131,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:16.461772+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:50:16.461772+0000","last_peered":"2023-10-04T01:50:16.461772+0000","last_clean":"2023-10-04T01:50:16.461772+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:50:16.461772+0000","last_undegraded":"2023-10-04T01:50:16.461772+0000","last_fullsized":"2023-10-04T01:50:16.461772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'518149","reported_seq":1021539,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:24.963732+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:50:24.963732+0000","last_peered":"2023-10-04T01:50:24.963732+0000","last_clean":"2023-10-04T01:50:24.963732+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:50:24.963732+0000","last_undegraded":"2023-10-04T01:50:24.963732+0000","last_fullsized":"2023-10-04T01:50:24.963732+0000","mapping_epoch":15,"log_start":"149'508066","ondisk_log_start":"149'508066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4325,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 676s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7180759040,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":508569,"num_read_kb":2003300,"num_write":507870,"num_write_kb":2031480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9650,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:16.480318+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:50:16.480318+0000","last_peered":"2023-10-04T01:50:16.480318+0000","last_clean":"2023-10-04T01:50:16.480318+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:50:16.480318+0000","last_undegraded":"2023-10-04T01:50:16.480318+0000","last_fullsized":"2023-10-04T01:50:16.480318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'492515","reported_seq":973516,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:22.216743+0000","last_change":"2023-10-04T01:50:21.507083+0000","last_active":"2023-10-04T01:50:22.216743+0000","last_peered":"2023-10-04T01:50:22.216743+0000","last_clean":"2023-10-04T01:50:22.216743+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:50:22.216743+0000","last_undegraded":"2023-10-04T01:50:22.216743+0000","last_fullsized":"2023-10-04T01:50:22.216743+0000","mapping_epoch":15,"log_start":"149'482463","ondisk_log_start":"149'482463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6859472915,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":486094,"num_read_kb":1913940,"num_write":486003,"num_write_kb":1944006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'18133","reported_seq":23598,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:21.927767+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:50:21.927767+0000","last_peered":"2023-10-04T01:50:21.927767+0000","last_clean":"2023-10-04T01:50:21.927767+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:50:21.927767+0000","last_undegraded":"2023-10-04T01:50:21.927767+0000","last_fullsized":"2023-10-04T01:50:21.927767+0000","mapping_epoch":12,"log_start":"85'8100","ondisk_log_start":"85'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":326,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:33.318906+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:49:33.318906+0000","last_peered":"2023-10-04T01:49:33.318906+0000","last_clean":"2023-10-04T01:49:33.318906+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:49:33.318906+0000","last_undegraded":"2023-10-04T01:49:33.318906+0000","last_fullsized":"2023-10-04T01:49:33.318906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27913117715,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1978145,"num_read_kb":7789772,"num_write":1976725,"num_write_kb":7906894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16189186048,"data_stored":35876282406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40269,"ondisk_log_size":40269,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57991,"ondisk_log_size":57991,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9667632,"kb_used_data":7904680,"kb_used_omap":2383,"kb_used_meta":1760560,"kb_avail":84704208,"statfs":{"total":96636764160,"available":86737108992,"internally_reserved":0,"allocated":8094392320,"data_stored":17936650874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1802814390},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":1.1539999999999999,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.61399999999999999,"15min":0.51100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":1.1910000000000001,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.31900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:49:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.624,"15min":0.46800000000000003},"min":{"1min":0.23699999999999999,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":2.6949999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.48299999999999998,"15min":0.47399999999999998},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.68200000000000005,"5min":4.5419999999999998,"15min":7.415},"last":0.23400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5690596,"kb_used_data":3890012,"kb_used_omap":999,"kb_used_meta":1799576,"kb_avail":88681244,"statfs":{"total":96636764160,"available":90809593856,"internally_reserved":0,"allocated":3983372288,"data_stored":8833128458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1842766604},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.33900000000000002,"15min":0.375},"min":{"1min":0.185,"5min":0.185,"15min":0.17899999999999999},"max":{"1min":0.39400000000000002,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.311},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.861,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.49099999999999999,"15min":0.65700000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":0.56100000000000005,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.58799999999999997,"15min":0.59799999999999998},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":1.466,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5900828,"kb_used_data":4021200,"kb_used_omap":1020,"kb_used_meta":1878595,"kb_avail":88471012,"statfs":{"total":96636764160,"available":90594316288,"internally_reserved":0,"allocated":4117708800,"data_stored":9108269921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1923681948},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,5,14,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:49:31 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.61299999999999999,"15min":1.234},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.46999999999999997,"5min":17.991,"15min":207.047},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.79100000000000004,"15min":1.323},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.80200000000000005,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:49:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.61299999999999999,"15min":0.56999999999999995},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22500000000000001},"max":{"1min":0.46300000000000002,"5min":14.118,"15min":14.118},"last":0.44},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.61099999999999999,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55400000000000005,"5min":14.051,"15min":14.051},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4115755008,"data_stored":9107730432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3980910592,"data_stored":8832483347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8092520448,"data_stored":17936068627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:50:26.232 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:26.232 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:26.232 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:26.232 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:26.233 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:50:46.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:50:46.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:50:46.856 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:50:46.878 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2475,"stamp":"2023-10-04T01:50:45.731063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27944542777,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2196387,"num_read_kb":7994853,"num_write":2039895,"num_write_kb":7993119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98402,"ondisk_log_size":98402,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21513652,"kb_used_data":15876188,"kb_used_omap":4402,"kb_used_meta":5633037,"kb_avail":261601868,"statfs":{"total":289910292480,"available":267880312832,"internally_reserved":0,"allocated":16257216512,"data_stored":35939792357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5768230238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,8,7,14,12,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19263488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4386,"num_read_kb":17516,"num_write":4700,"num_write_kb":18800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002063"},"pg_stats":[{"pgid":"2.7","version":"150'8379","reported_seq":11815,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:42.058760+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:50:42.058760+0000","last_peered":"2023-10-04T01:50:42.058760+0000","last_clean":"2023-10-04T01:50:42.058760+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:50:42.058760+0000","last_undegraded":"2023-10-04T01:50:42.058760+0000","last_fullsized":"2023-10-04T01:50:42.058760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8379,"log_dups_size":0,"ondisk_log_size":8379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'7442","reported_seq":11162,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:41.262416+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:50:41.262416+0000","last_peered":"2023-10-04T01:50:41.262416+0000","last_clean":"2023-10-04T01:50:41.262416+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:50:41.262416+0000","last_undegraded":"2023-10-04T01:50:41.262416+0000","last_fullsized":"2023-10-04T01:50:41.262416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7442,"log_dups_size":0,"ondisk_log_size":7442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'8523","reported_seq":12266,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:42.058765+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:50:42.058765+0000","last_peered":"2023-10-04T01:50:42.058765+0000","last_clean":"2023-10-04T01:50:42.058765+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:50:42.058765+0000","last_undegraded":"2023-10-04T01:50:42.058765+0000","last_fullsized":"2023-10-04T01:50:42.058765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'6952","reported_seq":11541,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:42.058925+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:50:42.058925+0000","last_peered":"2023-10-04T01:50:42.058925+0000","last_clean":"2023-10-04T01:50:42.058925+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:50:42.058925+0000","last_undegraded":"2023-10-04T01:50:42.058925+0000","last_fullsized":"2023-10-04T01:50:42.058925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'509453","reported_seq":1003419,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:45.183763+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:50:45.183763+0000","last_peered":"2023-10-04T01:50:45.183763+0000","last_clean":"2023-10-04T01:50:45.183763+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:50:45.183763+0000","last_undegraded":"2023-10-04T01:50:45.183763+0000","last_fullsized":"2023-10-04T01:50:45.183763+0000","mapping_epoch":15,"log_start":"149'499373","ondisk_log_start":"149'499373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4907,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 700s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7017717760,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":498970,"num_read_kb":1964800,"num_write":498854,"num_write_kb":1995416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'17211","reported_seq":22463,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:43.725217+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:50:43.725217+0000","last_peered":"2023-10-04T01:50:43.725217+0000","last_clean":"2023-10-04T01:50:43.725217+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:50:43.725217+0000","last_undegraded":"2023-10-04T01:50:43.725217+0000","last_fullsized":"2023-10-04T01:50:43.725217+0000","mapping_epoch":12,"log_start":"83'7200","ondisk_log_start":"83'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'498367","reported_seq":981529,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:42.352100+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:50:42.352100+0000","last_peered":"2023-10-04T01:50:42.352100+0000","last_clean":"2023-10-04T01:50:42.352100+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:50:42.352100+0000","last_undegraded":"2023-10-04T01:50:42.352100+0000","last_fullsized":"2023-10-04T01:50:42.352100+0000","mapping_epoch":15,"log_start":"149'488267","ondisk_log_start":"149'488267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5087,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 695s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6870425600,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":488061,"num_read_kb":1921904,"num_write":487726,"num_write_kb":1950904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9148,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:36.942412+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:50:36.942412+0000","last_peered":"2023-10-04T01:50:36.942412+0000","last_clean":"2023-10-04T01:50:36.942412+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:50:36.942412+0000","last_undegraded":"2023-10-04T01:50:36.942412+0000","last_fullsized":"2023-10-04T01:50:36.942412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'520162","reported_seq":1025358,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:50:45.184379+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:50:45.184379+0000","last_peered":"2023-10-04T01:50:45.184379+0000","last_clean":"2023-10-04T01:50:45.184379+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:50:45.184379+0000","last_undegraded":"2023-10-04T01:50:45.184379+0000","last_fullsized":"2023-10-04T01:50:45.184379+0000","mapping_epoch":15,"log_start":"149'510066","ondisk_log_start":"149'510066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4441,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 697s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7188615168,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":510367,"num_read_kb":2010480,"num_write":509788,"num_write_kb":2039152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9662,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:36.940421+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:50:36.940421+0000","last_peered":"2023-10-04T01:50:36.940421+0000","last_clean":"2023-10-04T01:50:36.940421+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:50:36.940421+0000","last_undegraded":"2023-10-04T01:50:36.940421+0000","last_fullsized":"2023-10-04T01:50:36.940421+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'494395","reported_seq":977120,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:42.358759+0000","last_change":"2023-10-04T01:50:39.763089+0000","last_active":"2023-10-04T01:50:42.358759+0000","last_peered":"2023-10-04T01:50:42.358759+0000","last_clean":"2023-10-04T01:50:42.358759+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:50:42.358759+0000","last_undegraded":"2023-10-04T01:50:42.358759+0000","last_fullsized":"2023-10-04T01:50:42.358759+0000","mapping_epoch":15,"log_start":"149'484363","ondisk_log_start":"149'484363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6867177491,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":487815,"num_read_kb":1920816,"num_write":487883,"num_write_kb":1951526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'18157","reported_seq":23634,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:50:42.059359+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:50:42.059359+0000","last_peered":"2023-10-04T01:50:42.059359+0000","last_clean":"2023-10-04T01:50:42.059359+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:50:42.059359+0000","last_undegraded":"2023-10-04T01:50:42.059359+0000","last_fullsized":"2023-10-04T01:50:42.059359+0000","mapping_epoch":12,"log_start":"85'8100","ondisk_log_start":"85'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":326,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:33.318906+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:49:33.318906+0000","last_peered":"2023-10-04T01:49:33.318906+0000","last_clean":"2023-10-04T01:49:33.318906+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:49:33.318906+0000","last_undegraded":"2023-10-04T01:49:33.318906+0000","last_fullsized":"2023-10-04T01:49:33.318906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27943936019,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1985213,"num_read_kb":7818000,"num_write":1984251,"num_write_kb":7936998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16250585088,"data_stored":35937681446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40308,"ondisk_log_size":40308,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":57995,"ondisk_log_size":57995,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739300,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9787504,"kb_used_data":7934376,"kb_used_omap":2383,"kb_used_meta":1850736,"kb_avail":84584336,"statfs":{"total":96636764160,"available":86614360064,"internally_reserved":0,"allocated":8124801024,"data_stored":17967059578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1895154614},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.54700000000000004,"15min":0.44700000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":1.1539999999999999,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.61399999999999999,"15min":0.51100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":1.1910000000000001,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.52500000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:49:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.624,"15min":0.46800000000000003},"min":{"1min":0.23699999999999999,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":2.6949999999999998,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.48299999999999998,"15min":0.47399999999999998},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.68200000000000005,"5min":4.5419999999999998,"15min":7.415},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5755456,"kb_used_data":3904888,"kb_used_omap":999,"kb_used_meta":1849560,"kb_avail":88616384,"statfs":{"total":96636764160,"available":90743177216,"internally_reserved":0,"allocated":3998605312,"data_stored":8848361482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1893950220},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,0,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.33900000000000002,"15min":0.375},"min":{"1min":0.185,"5min":0.185,"15min":0.17899999999999999},"max":{"1min":0.39400000000000002,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.52500000000000002,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.861,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:49:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.49099999999999999,"15min":0.65700000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":0.56100000000000005,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.58799999999999997,"15min":0.59799999999999998},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":1.466,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5970692,"kb_used_data":4036924,"kb_used_omap":1020,"kb_used_meta":1932739,"kb_avail":88401148,"statfs":{"total":96636764160,"available":90522775552,"internally_reserved":0,"allocated":4133810176,"data_stored":9124371297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":1979125404},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,6,4,9,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:50:31 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":1.24},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.8600000000000001,"5min":17.991,"15min":207.047},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":1.349},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.6210000000000004,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.39600000000000002,"15min":0.57899999999999996},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":1.915,"5min":1.915,"15min":14.118},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.42799999999999999,"15min":0.49199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":3.0720000000000001,"15min":14.051},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4131405824,"data_stored":9123381248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3996090368,"data_stored":8847663123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8123088896,"data_stored":17966637075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:50:46.881 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:46.881 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:46.881 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:46.882 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:50:46.882 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:51:06.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:51:07.503 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:51:07.504 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:51:07.549 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2486,"stamp":"2023-10-04T01:51:05.734519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27974136377,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2203531,"num_read_kb":8023333,"num_write":2047105,"num_write_kb":8021959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98482,"ondisk_log_size":98482,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21757996,"kb_used_data":15934100,"kb_used_omap":4402,"kb_used_meta":5819469,"kb_avail":261357524,"statfs":{"total":289910292480,"available":267630104576,"internally_reserved":0,"allocated":16316518400,"data_stored":35999082590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":5959136606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,3,4,6,4,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18325504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4422,"num_read_kb":17636,"num_write":4464,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002062"},"pg_stats":[{"pgid":"2.7","version":"151'8401","reported_seq":11844,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:01.771117+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:51:01.771117+0000","last_peered":"2023-10-04T01:51:01.771117+0000","last_clean":"2023-10-04T01:51:01.771117+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:51:01.771117+0000","last_undegraded":"2023-10-04T01:51:01.771117+0000","last_fullsized":"2023-10-04T01:51:01.771117+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'7453","reported_seq":11177,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:01.079793+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:51:01.079793+0000","last_peered":"2023-10-04T01:51:01.079793+0000","last_clean":"2023-10-04T01:51:01.079793+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:51:01.079793+0000","last_undegraded":"2023-10-04T01:51:01.079793+0000","last_fullsized":"2023-10-04T01:51:01.079793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'8545","reported_seq":12296,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:01.774003+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:51:01.774003+0000","last_peered":"2023-10-04T01:51:01.774003+0000","last_clean":"2023-10-04T01:51:01.774003+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:51:01.774003+0000","last_undegraded":"2023-10-04T01:51:01.774003+0000","last_fullsized":"2023-10-04T01:51:01.774003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8545,"log_dups_size":0,"ondisk_log_size":8545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'6974","reported_seq":11571,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:01.769058+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:51:01.769058+0000","last_peered":"2023-10-04T01:51:01.769058+0000","last_clean":"2023-10-04T01:51:01.769058+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:51:01.769058+0000","last_undegraded":"2023-10-04T01:51:01.769058+0000","last_fullsized":"2023-10-04T01:51:01.769058+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'511396","reported_seq":1007162,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:05.281721+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:51:05.281721+0000","last_peered":"2023-10-04T01:51:05.281721+0000","last_clean":"2023-10-04T01:51:05.281721+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:51:05.281721+0000","last_undegraded":"2023-10-04T01:51:05.281721+0000","last_fullsized":"2023-10-04T01:51:05.281721+0000","mapping_epoch":15,"log_start":"149'501373","ondisk_log_start":"149'501373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5103,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 721s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7025004544,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":500754,"num_read_kb":1971904,"num_write":500635,"num_write_kb":2002540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'17262","reported_seq":22528,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:04.015001+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:51:04.015001+0000","last_peered":"2023-10-04T01:51:04.015001+0000","last_clean":"2023-10-04T01:51:04.015001+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:51:04.015001+0000","last_undegraded":"2023-10-04T01:51:04.015001+0000","last_fullsized":"2023-10-04T01:51:04.015001+0000","mapping_epoch":12,"log_start":"83'7200","ondisk_log_start":"83'7200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'500260","reported_seq":985219,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:02.453444+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:51:02.453444+0000","last_peered":"2023-10-04T01:51:02.453444+0000","last_clean":"2023-10-04T01:51:02.453444+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:51:02.453444+0000","last_undegraded":"2023-10-04T01:51:02.453444+0000","last_fullsized":"2023-10-04T01:51:02.453444+0000","mapping_epoch":15,"log_start":"149'490167","ondisk_log_start":"149'490167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5249,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 715s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6877679616,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":489845,"num_read_kb":1929024,"num_write":489485,"num_write_kb":1957940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9165,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:50:57.445318+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:50:57.445318+0000","last_peered":"2023-10-04T01:50:57.445318+0000","last_clean":"2023-10-04T01:50:57.445318+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:50:57.445318+0000","last_undegraded":"2023-10-04T01:50:57.445318+0000","last_fullsized":"2023-10-04T01:50:57.445318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'522136","reported_seq":1029193,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:05.279408+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:51:05.279408+0000","last_peered":"2023-10-04T01:51:05.279408+0000","last_clean":"2023-10-04T01:51:05.279408+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:51:05.279408+0000","last_undegraded":"2023-10-04T01:51:05.279408+0000","last_fullsized":"2023-10-04T01:51:05.279408+0000","mapping_epoch":15,"log_start":"149'512066","ondisk_log_start":"149'512066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4579,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 717s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7196258304,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":512218,"num_read_kb":2017864,"num_write":511649,"num_write_kb":2046596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9675,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:50:57.421591+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:50:57.421591+0000","last_peered":"2023-10-04T01:50:57.421591+0000","last_clean":"2023-10-04T01:50:57.421591+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:50:57.421591+0000","last_undegraded":"2023-10-04T01:50:57.421591+0000","last_fullsized":"2023-10-04T01:50:57.421591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'496204","reported_seq":980665,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:02.444388+0000","last_change":"2023-10-04T01:50:58.732540+0000","last_active":"2023-10-04T01:51:02.444388+0000","last_peered":"2023-10-04T01:51:02.444388+0000","last_clean":"2023-10-04T01:51:02.444388+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:51:02.444388+0000","last_undegraded":"2023-10-04T01:51:02.444388+0000","last_fullsized":"2023-10-04T01:51:02.444388+0000","mapping_epoch":15,"log_start":"149'486163","ondisk_log_start":"149'486163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6874587155,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":489540,"num_read_kb":1927688,"num_write":489692,"num_write_kb":1958762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'18190","reported_seq":23678,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:01.817358+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:51:01.817358+0000","last_peered":"2023-10-04T01:51:01.817358+0000","last_clean":"2023-10-04T01:51:01.817358+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:51:01.817358+0000","last_undegraded":"2023-10-04T01:51:01.817358+0000","last_fullsized":"2023-10-04T01:51:01.817358+0000","mapping_epoch":12,"log_start":"85'8100","ondisk_log_start":"85'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":327,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:42.116213+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:49:42.116213+0000","last_peered":"2023-10-04T01:49:42.116213+0000","last_clean":"2023-10-04T01:49:42.116213+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:49:42.116213+0000","last_undegraded":"2023-10-04T01:49:42.116213+0000","last_fullsized":"2023-10-04T01:49:42.116213+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":27973529619,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":1992357,"num_read_kb":7846480,"num_write":1991461,"num_write_kb":7965838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16309878784,"data_stored":35996975142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58156,"ondisk_log_size":58156,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9912648,"kb_used_data":7963328,"kb_used_omap":2383,"kb_used_meta":1946928,"kb_avail":84459192,"statfs":{"total":96636764160,"available":86486212608,"internally_reserved":0,"allocated":8154447872,"data_stored":17996702541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":1993655222},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.52200000000000002,"15min":0.441},"min":{"1min":0.17799999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.52700000000000002,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.183},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.58699999999999997,"15min":0.502},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":0.60599999999999998,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.24399999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:50:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.47499999999999998},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":3.1960000000000002,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.58299999999999996,"5min":4.5419999999999998,"15min":7.415},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739302,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5805776,"kb_used_data":3919112,"kb_used_omap":999,"kb_used_meta":1885656,"kb_avail":88566064,"statfs":{"total":96636764160,"available":90691649536,"internally_reserved":0,"allocated":4013170688,"data_stored":8862922973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1930912524},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.32100000000000001,"15min":0.376},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.60299999999999998,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.53200000000000003,"15min":0.505},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.5519999999999996,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49299999999999999,"15min":0.63500000000000001},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":2.0049999999999999,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.52900000000000003,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6039572,"kb_used_data":4051660,"kb_used_omap":1020,"kb_used_meta":1986883,"kb_avail":88332268,"statfs":{"total":96636764160,"available":90452242432,"internally_reserved":0,"allocated":4148899840,"data_stored":9139457076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":2034568860},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:50:31 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":1.24},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.8600000000000001,"5min":17.991,"15min":207.047},"last":0.308},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":1.349},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.6210000000000004,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.39600000000000002,"15min":0.57899999999999996},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":1.915,"5min":1.915,"15min":14.118},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.42799999999999999,"15min":0.49199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":3.0720000000000001,"15min":14.051},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4146319360,"data_stored":9138294784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4010708992,"data_stored":8862281747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8152850432,"data_stored":17996398611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:51:07.551 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:07.551 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:07.551 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:07.552 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:07.552 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:51:18.192 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.183+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:51:18.206 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:51:18.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:51:18.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:51:18.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:51:18.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:51:18.207 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:51:18.208 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:51:18.208 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:51:18.195+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:51:27.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:51:28.109 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:51:28.110 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:51:28.144 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2497,"stamp":"2023-10-04T01:51:27.738240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28007854649,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2211757,"num_read_kb":8056161,"num_write":2055305,"num_write_kb":8054759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98430,"ondisk_log_size":98430,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21944552,"kb_used_data":16002768,"kb_used_omap":4402,"kb_used_meta":5937357,"kb_avail":261170968,"statfs":{"total":289910292480,"available":267439071232,"internally_reserved":0,"allocated":16386834432,"data_stored":36069398622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4508322,"internal_metadata":6079853918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,10,12,13,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19496960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4809,"num_read_kb":19192,"num_write":4729,"num_write_kb":18916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002015"},"pg_stats":[{"pgid":"2.7","version":"151'8425","reported_seq":11878,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:25.822586+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:51:25.822586+0000","last_peered":"2023-10-04T01:51:25.822586+0000","last_clean":"2023-10-04T01:51:25.822586+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:51:25.822586+0000","last_undegraded":"2023-10-04T01:51:25.822586+0000","last_fullsized":"2023-10-04T01:51:25.822586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'7465","reported_seq":11194,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:25.500535+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:51:25.500535+0000","last_peered":"2023-10-04T01:51:25.500535+0000","last_clean":"2023-10-04T01:51:25.500535+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:51:25.500535+0000","last_undegraded":"2023-10-04T01:51:25.500535+0000","last_fullsized":"2023-10-04T01:51:25.500535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'8569","reported_seq":12330,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:25.821972+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:51:25.821972+0000","last_peered":"2023-10-04T01:51:25.821972+0000","last_clean":"2023-10-04T01:51:25.821972+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:51:25.821972+0000","last_undegraded":"2023-10-04T01:51:25.821972+0000","last_fullsized":"2023-10-04T01:51:25.821972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8569,"log_dups_size":0,"ondisk_log_size":8569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'6998","reported_seq":11605,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:25.818321+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:51:25.818321+0000","last_peered":"2023-10-04T01:51:25.818321+0000","last_clean":"2023-10-04T01:51:25.818321+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:51:25.818321+0000","last_undegraded":"2023-10-04T01:51:25.818321+0000","last_fullsized":"2023-10-04T01:51:25.818321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'513311","reported_seq":1010930,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:25.450289+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:51:25.450289+0000","last_peered":"2023-10-04T01:51:25.450289+0000","last_clean":"2023-10-04T01:51:25.450289+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:51:25.450289+0000","last_undegraded":"2023-10-04T01:51:25.450289+0000","last_fullsized":"2023-10-04T01:51:25.450289+0000","mapping_epoch":15,"log_start":"150'503273","ondisk_log_start":"150'503273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5214,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 741s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7032516608,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":502595,"num_read_kb":1979260,"num_write":502461,"num_write_kb":2009844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'17304","reported_seq":22582,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:24.222683+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:51:24.222683+0000","last_peered":"2023-10-04T01:51:24.222683+0000","last_clean":"2023-10-04T01:51:24.222683+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:51:24.222683+0000","last_undegraded":"2023-10-04T01:51:24.222683+0000","last_fullsized":"2023-10-04T01:51:24.222683+0000","mapping_epoch":12,"log_start":"83'7300","ondisk_log_start":"83'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'502651","reported_seq":989842,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:27.558895+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:51:27.558895+0000","last_peered":"2023-10-04T01:51:27.558895+0000","last_clean":"2023-10-04T01:51:27.558895+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:51:27.558895+0000","last_undegraded":"2023-10-04T01:51:27.558895+0000","last_fullsized":"2023-10-04T01:51:27.558895+0000","mapping_epoch":15,"log_start":"150'492567","ondisk_log_start":"150'492567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5420,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 740s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6886952960,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":492066,"num_read_kb":1937880,"num_write":491735,"num_write_kb":1966940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9185,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:23.032254+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:51:23.032254+0000","last_peered":"2023-10-04T01:51:23.032254+0000","last_clean":"2023-10-04T01:51:23.032254+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:51:23.032254+0000","last_undegraded":"2023-10-04T01:51:23.032254+0000","last_fullsized":"2023-10-04T01:51:23.032254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'524113","reported_seq":1033042,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:25.446695+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:51:25.446695+0000","last_peered":"2023-10-04T01:51:25.446695+0000","last_clean":"2023-10-04T01:51:25.446695+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:51:25.446695+0000","last_undegraded":"2023-10-04T01:51:25.446695+0000","last_fullsized":"2023-10-04T01:51:25.446695+0000","mapping_epoch":15,"log_start":"150'514066","ondisk_log_start":"150'514066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4646,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 737s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7204134912,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":514078,"num_read_kb":2025288,"num_write":513570,"num_write_kb":2054280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9690,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:23.170403+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:51:23.170403+0000","last_peered":"2023-10-04T01:51:23.170403+0000","last_clean":"2023-10-04T01:51:23.170403+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:51:23.170403+0000","last_undegraded":"2023-10-04T01:51:23.170403+0000","last_fullsized":"2023-10-04T01:51:23.170403+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'498407","reported_seq":985178,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:27.570551+0000","last_change":"2023-10-04T01:51:26.925134+0000","last_active":"2023-10-04T01:51:27.570551+0000","last_peered":"2023-10-04T01:51:27.570551+0000","last_clean":"2023-10-04T01:51:27.570551+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:51:27.570551+0000","last_undegraded":"2023-10-04T01:51:27.570551+0000","last_fullsized":"2023-10-04T01:51:27.570551+0000","mapping_epoch":15,"log_start":"150'488363","ondisk_log_start":"150'488363","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6883643411,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":491844,"num_read_kb":1936880,"num_write":491895,"num_write_kb":1967574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'18226","reported_seq":23729,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:25.898015+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:51:25.898015+0000","last_peered":"2023-10-04T01:51:25.898015+0000","last_clean":"2023-10-04T01:51:25.898015+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:51:25.898015+0000","last_undegraded":"2023-10-04T01:51:25.898015+0000","last_fullsized":"2023-10-04T01:51:25.898015+0000","mapping_epoch":12,"log_start":"87'8200","ondisk_log_start":"87'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":327,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-04T01:49:42.116213+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:49:42.116213+0000","last_peered":"2023-10-04T01:49:42.116213+0000","last_clean":"2023-10-04T01:49:42.116213+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:49:42.116213+0000","last_undegraded":"2023-10-04T01:49:42.116213+0000","last_fullsized":"2023-10-04T01:49:42.116213+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28007247891,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2000583,"num_read_kb":7879308,"num_write":1999661,"num_write_kb":7998638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16380739584,"data_stored":36067835942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58118,"ondisk_log_size":58118,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9989256,"kb_used_data":7999808,"kb_used_omap":2383,"kb_used_meta":1987056,"kb_avail":84382584,"statfs":{"total":96636764160,"available":86407766016,"internally_reserved":0,"allocated":8191803392,"data_stored":18034058061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":2034746294},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.52200000000000002,"15min":0.441},"min":{"1min":0.17799999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.52700000000000002,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.58699999999999997,"15min":0.502},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":0.60599999999999998,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.30599999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:50:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.47499999999999998},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":3.1960000000000002,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.314,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.58299999999999996,"5min":4.5419999999999998,"15min":7.415},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739307,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5865280,"kb_used_data":3936952,"kb_used_omap":999,"kb_used_meta":1927320,"kb_avail":88506560,"statfs":{"total":96636764160,"available":90630717440,"internally_reserved":0,"allocated":4031438848,"data_stored":8881191133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":1973576460},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.32100000000000001,"15min":0.376},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.60299999999999998,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.53200000000000003,"15min":0.505},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.5519999999999996,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.29199999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49299999999999999,"15min":0.63500000000000001},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":2.0049999999999999,"5min":5.9790000000000001,"15min":10.494999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.52900000000000003,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359739307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6090016,"kb_used_data":4066008,"kb_used_omap":1020,"kb_used_meta":2022979,"kb_avail":88281824,"statfs":{"total":96636764160,"available":90400587776,"internally_reserved":0,"allocated":4163592192,"data_stored":9154149428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1044836,"internal_metadata":2071531164},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,4,4,7,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:50:31 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":1.24},"min":{"1min":0.23799999999999999,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.8600000000000001,"5min":17.991,"15min":207.047},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":1.349},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.6210000000000004,"5min":17.957000000000001,"15min":207.00200000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.39600000000000002,"15min":0.57899999999999996},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":1.915,"5min":1.915,"15min":14.118},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.42799999999999999,"15min":0.49199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":3.0720000000000001,"15min":14.051},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4161667072,"data_stored":9153642496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4028944384,"data_stored":8880517139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8190128128,"data_stored":18033676307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:51:28.145 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:28.146 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:28.146 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:28.146 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:28.146 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:51:48.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:51:48.849 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:51:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:51:48.879 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2507,"stamp":"2023-10-04T01:51:47.741643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28038156857,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2219284,"num_read_kb":8086189,"num_write":2062669,"num_write_kb":8084215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98592,"ondisk_log_size":98592,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22115168,"kb_used_data":16061448,"kb_used_omap":4739,"kb_used_meta":6048956,"kb_avail":261000352,"statfs":{"total":289910292480,"available":267264360448,"internally_reserved":0,"allocated":16446922752,"data_stored":36129486942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4853186,"internal_metadata":6194131518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,4,11,9,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19054592,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4651,"num_read_kb":18564,"num_write":4634,"num_write_kb":18536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002029"},"pg_stats":[{"pgid":"2.7","version":"151'8451","reported_seq":11913,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:44.824273+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:51:44.824273+0000","last_peered":"2023-10-04T01:51:44.824273+0000","last_clean":"2023-10-04T01:51:44.824273+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:51:44.824273+0000","last_undegraded":"2023-10-04T01:51:44.824273+0000","last_fullsized":"2023-10-04T01:51:44.824273+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8451,"log_dups_size":0,"ondisk_log_size":8451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'7478","reported_seq":11212,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:44.746525+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:51:44.746525+0000","last_peered":"2023-10-04T01:51:44.746525+0000","last_clean":"2023-10-04T01:51:44.746525+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:51:44.746525+0000","last_undegraded":"2023-10-04T01:51:44.746525+0000","last_fullsized":"2023-10-04T01:51:44.746525+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'8595","reported_seq":12364,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:44.806555+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:51:44.806555+0000","last_peered":"2023-10-04T01:51:44.806555+0000","last_clean":"2023-10-04T01:51:44.806555+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:51:44.806555+0000","last_undegraded":"2023-10-04T01:51:44.806555+0000","last_fullsized":"2023-10-04T01:51:44.806555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8595,"log_dups_size":0,"ondisk_log_size":8595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'7024","reported_seq":11639,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:44.817917+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:51:44.817917+0000","last_peered":"2023-10-04T01:51:44.817917+0000","last_clean":"2023-10-04T01:51:44.817917+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:51:44.817917+0000","last_undegraded":"2023-10-04T01:51:44.817917+0000","last_fullsized":"2023-10-04T01:51:44.817917+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7024,"log_dups_size":0,"ondisk_log_size":7024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'515281","reported_seq":1014797,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:45.657104+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:51:45.657104+0000","last_peered":"2023-10-04T01:51:45.657104+0000","last_clean":"2023-10-04T01:51:45.657104+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:51:45.657104+0000","last_undegraded":"2023-10-04T01:51:45.657104+0000","last_fullsized":"2023-10-04T01:51:45.657104+0000","mapping_epoch":15,"log_start":"150'505273","ondisk_log_start":"150'505273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5332,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 761s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7040286720,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":504484,"num_read_kb":1986800,"num_write":504335,"num_write_kb":2017340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"151'17349","reported_seq":22639,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:43.837124+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:51:43.837124+0000","last_peered":"2023-10-04T01:51:43.837124+0000","last_clean":"2023-10-04T01:51:43.837124+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:51:43.837124+0000","last_undegraded":"2023-10-04T01:51:43.837124+0000","last_fullsized":"2023-10-04T01:51:43.837124+0000","mapping_epoch":12,"log_start":"83'7300","ondisk_log_start":"83'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'504575","reported_seq":993573,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:47.662069+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:51:47.662069+0000","last_peered":"2023-10-04T01:51:47.662069+0000","last_clean":"2023-10-04T01:51:47.662069+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:51:47.662069+0000","last_undegraded":"2023-10-04T01:51:47.662069+0000","last_fullsized":"2023-10-04T01:51:47.662069+0000","mapping_epoch":15,"log_start":"150'494567","ondisk_log_start":"150'494567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5552,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 760s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6894428160,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":493859,"num_read_kb":1945040,"num_write":493551,"num_write_kb":1974204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9201,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:43.518311+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:51:43.518311+0000","last_peered":"2023-10-04T01:51:43.518311+0000","last_clean":"2023-10-04T01:51:43.518311+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:51:43.518311+0000","last_undegraded":"2023-10-04T01:51:43.518311+0000","last_fullsized":"2023-10-04T01:51:43.518311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'526056","reported_seq":1036971,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:51:45.660256+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:51:45.660256+0000","last_peered":"2023-10-04T01:51:45.660256+0000","last_clean":"2023-10-04T01:51:45.660256+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:51:45.660256+0000","last_undegraded":"2023-10-04T01:51:45.660256+0000","last_fullsized":"2023-10-04T01:51:45.660256+0000","mapping_epoch":15,"log_start":"150'515966","ondisk_log_start":"150'515966","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4791,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 757s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7211614208,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":516057,"num_read_kb":2033192,"num_write":515394,"num_write_kb":2061576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9703,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:43.507175+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:51:43.507175+0000","last_peered":"2023-10-04T01:51:43.507175+0000","last_clean":"2023-10-04T01:51:43.507175+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:51:43.507175+0000","last_undegraded":"2023-10-04T01:51:43.507175+0000","last_fullsized":"2023-10-04T01:51:43.507175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'500257","reported_seq":988905,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:47.654864+0000","last_change":"2023-10-04T01:51:43.112386+0000","last_active":"2023-10-04T01:51:47.654864+0000","last_peered":"2023-10-04T01:51:47.654864+0000","last_clean":"2023-10-04T01:51:47.654864+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:51:47.654864+0000","last_undegraded":"2023-10-04T01:51:47.654864+0000","last_fullsized":"2023-10-04T01:51:47.654864+0000","mapping_epoch":15,"log_start":"150'490163","ondisk_log_start":"150'490163","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6891221011,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":493710,"num_read_kb":1944304,"num_write":493745,"num_write_kb":1974974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'18265","reported_seq":23780,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:44.847727+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:51:44.847727+0000","last_peered":"2023-10-04T01:51:44.847727+0000","last_clean":"2023-10-04T01:51:44.847727+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:51:44.847727+0000","last_undegraded":"2023-10-04T01:51:44.847727+0000","last_fullsized":"2023-10-04T01:51:44.847727+0000","mapping_epoch":12,"log_start":"87'8200","ondisk_log_start":"87'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":328,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:50:53.359154+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:50:53.359154+0000","last_peered":"2023-10-04T01:50:53.359154+0000","last_clean":"2023-10-04T01:50:53.359154+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:50:53.359154+0000","last_undegraded":"2023-10-04T01:50:53.359154+0000","last_fullsized":"2023-10-04T01:50:53.359154+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28037550099,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2008110,"num_read_kb":7909336,"num_write":2007025,"num_write_kb":8028094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16440934400,"data_stored":36128030758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58293,"ondisk_log_size":58293,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10131212,"kb_used_data":8028804,"kb_used_omap":2383,"kb_used_meta":2100016,"kb_avail":84240628,"statfs":{"total":96636764160,"available":86262403072,"internally_reserved":0,"allocated":8221495296,"data_stored":18063749965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2440266,"internal_metadata":2150417334},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.52200000000000002,"15min":0.441},"min":{"1min":0.17799999999999999,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.52700000000000002,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.38},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.58699999999999997,"15min":0.502},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.22900000000000001},"max":{"1min":0.60599999999999998,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:50:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.47499999999999998},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":3.1960000000000002,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.314,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.58299999999999996,"5min":4.5419999999999998,"15min":7.415},"last":0.51700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739311,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5968856,"kb_used_data":3951696,"kb_used_omap":999,"kb_used_meta":2016152,"kb_avail":88402984,"statfs":{"total":96636764160,"available":90524655616,"internally_reserved":0,"allocated":4046536704,"data_stored":8896288989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":2064540428},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.32100000000000001,"15min":0.376},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.60299999999999998,"5min":3.1850000000000001,"15min":5.7240000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.53200000000000003,"15min":0.505},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.5519999999999996,"5min":5.5730000000000004,"15min":5.6520000000000001},"last":0.88700000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:50:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49299999999999999,"15min":0.63500000000000001},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":2.0049999999999999,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":0.52900000000000003,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.39400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6015100,"kb_used_data":4080948,"kb_used_omap":1357,"kb_used_meta":1932786,"kb_avail":88356740,"statfs":{"total":96636764160,"available":90477301760,"internally_reserved":0,"allocated":4178890752,"data_stored":9169447988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1979173756},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,4,3,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:51:32 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36299999999999999,"15min":1.2390000000000001},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":0.69199999999999995,"5min":1.8600000000000001,"15min":207.047},"last":0.441},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":1.3560000000000001},"min":{"1min":0.26100000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.859,"5min":5.6210000000000004,"15min":207.00200000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:50:43 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.39600000000000002,"15min":0.57899999999999996},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":1.915,"5min":1.915,"15min":14.118},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.42799999999999999,"15min":0.49199999999999999},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.57499999999999996,"5min":3.0720000000000001,"15min":14.051},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4176814080,"data_stored":9168789504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4043964416,"data_stored":8895537171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8220155904,"data_stored":18063704083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:51:48.881 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:48.881 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:48.882 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:48.882 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:51:48.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-04T01:51:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-04T01:51:49.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-04T01:51:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-04T01:51:49.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-04T01:51:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-04T01:51:50.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:51:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:51:51.142 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:52:11.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:52:11.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:52:11.754 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:52:11.794 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2519,"stamp":"2023-10-04T01:52:09.745403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28067885625,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2226732,"num_read_kb":8115893,"num_write":2069865,"num_write_kb":8112999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98661,"ondisk_log_size":98661,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21605252,"kb_used_data":16119080,"kb_used_omap":4994,"kb_used_meta":5481149,"kb_avail":261510268,"statfs":{"total":289910292480,"available":267786514432,"internally_reserved":0,"allocated":16505937920,"data_stored":36188490326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5114626,"internal_metadata":5612696830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,4,4,5,16,16,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15167488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3785,"num_read_kb":15084,"num_write":3670,"num_write_kb":14680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001673"},"pg_stats":[{"pgid":"2.7","version":"152'8467","reported_seq":11938,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:05.273105+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:52:05.273105+0000","last_peered":"2023-10-04T01:52:05.273105+0000","last_clean":"2023-10-04T01:52:05.273105+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:52:05.273105+0000","last_undegraded":"2023-10-04T01:52:05.273105+0000","last_fullsized":"2023-10-04T01:52:05.273105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'7486","reported_seq":11225,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:04.973581+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:52:04.973581+0000","last_peered":"2023-10-04T01:52:04.973581+0000","last_clean":"2023-10-04T01:52:04.973581+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:52:04.973581+0000","last_undegraded":"2023-10-04T01:52:04.973581+0000","last_fullsized":"2023-10-04T01:52:04.973581+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'8611","reported_seq":12389,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:05.270089+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:52:05.270089+0000","last_peered":"2023-10-04T01:52:05.270089+0000","last_clean":"2023-10-04T01:52:05.270089+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:52:05.270089+0000","last_undegraded":"2023-10-04T01:52:05.270089+0000","last_fullsized":"2023-10-04T01:52:05.270089+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'7040","reported_seq":11664,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:05.272074+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:52:05.272074+0000","last_peered":"2023-10-04T01:52:05.272074+0000","last_clean":"2023-10-04T01:52:05.272074+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:52:05.272074+0000","last_undegraded":"2023-10-04T01:52:05.272074+0000","last_fullsized":"2023-10-04T01:52:05.272074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7040,"log_dups_size":0,"ondisk_log_size":7040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'517222","reported_seq":1018584,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:05.849458+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:52:05.849458+0000","last_peered":"2023-10-04T01:52:05.849458+0000","last_clean":"2023-10-04T01:52:05.849458+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:52:05.849458+0000","last_undegraded":"2023-10-04T01:52:05.849458+0000","last_fullsized":"2023-10-04T01:52:05.849458+0000","mapping_epoch":15,"log_start":"150'507173","ondisk_log_start":"150'507173","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5475,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 781s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7047827456,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":506317,"num_read_kb":1994108,"num_write":506161,"num_write_kb":2024644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'17391","reported_seq":22694,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:04.051426+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:52:04.051426+0000","last_peered":"2023-10-04T01:52:04.051426+0000","last_clean":"2023-10-04T01:52:04.051426+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:52:04.051426+0000","last_undegraded":"2023-10-04T01:52:04.051426+0000","last_fullsized":"2023-10-04T01:52:04.051426+0000","mapping_epoch":12,"log_start":"83'7300","ondisk_log_start":"83'7300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'506476","reported_seq":997352,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:07.765697+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:52:07.765697+0000","last_peered":"2023-10-04T01:52:07.765697+0000","last_clean":"2023-10-04T01:52:07.765697+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:52:07.765697+0000","last_undegraded":"2023-10-04T01:52:07.765697+0000","last_fullsized":"2023-10-04T01:52:07.765697+0000","mapping_epoch":15,"log_start":"150'496467","ondisk_log_start":"150'496467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5698,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 780s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6901809152,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":495726,"num_read_kb":1952476,"num_write":495332,"num_write_kb":1981328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9218,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:03.992339+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:52:03.992339+0000","last_peered":"2023-10-04T01:52:03.992339+0000","last_clean":"2023-10-04T01:52:03.992339+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:52:03.992339+0000","last_undegraded":"2023-10-04T01:52:03.992339+0000","last_fullsized":"2023-10-04T01:52:03.992339+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'528000","reported_seq":1040801,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:05.860706+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:52:05.860706+0000","last_peered":"2023-10-04T01:52:05.860706+0000","last_clean":"2023-10-04T01:52:05.860706+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:52:05.860706+0000","last_undegraded":"2023-10-04T01:52:05.860706+0000","last_fullsized":"2023-10-04T01:52:05.860706+0000","mapping_epoch":15,"log_start":"150'517966","ondisk_log_start":"150'517966","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4935,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 777s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7219183616,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":517930,"num_read_kb":2040660,"num_write":517222,"num_write_kb":2068888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9716,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:03.979672+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:52:03.979672+0000","last_peered":"2023-10-04T01:52:03.979672+0000","last_clean":"2023-10-04T01:52:03.979672+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:52:03.979672+0000","last_undegraded":"2023-10-04T01:52:03.979672+0000","last_fullsized":"2023-10-04T01:52:03.979672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'502018","reported_seq":992551,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:07.771599+0000","last_change":"2023-10-04T01:52:06.100832+0000","last_active":"2023-10-04T01:52:07.771599+0000","last_peered":"2023-10-04T01:52:07.771599+0000","last_clean":"2023-10-04T01:52:07.771599+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:52:07.771599+0000","last_undegraded":"2023-10-04T01:52:07.771599+0000","last_fullsized":"2023-10-04T01:52:07.771599+0000","mapping_epoch":15,"log_start":"150'491963","ondisk_log_start":"150'491963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6898458643,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":495585,"num_read_kb":1951796,"num_write":495506,"num_write_kb":1982018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'18289","reported_seq":23817,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:05.322894+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:52:05.322894+0000","last_peered":"2023-10-04T01:52:05.322894+0000","last_clean":"2023-10-04T01:52:05.322894+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:52:05.322894+0000","last_undegraded":"2023-10-04T01:52:05.322894+0000","last_fullsized":"2023-10-04T01:52:05.322894+0000","mapping_epoch":12,"log_start":"87'8200","ondisk_log_start":"87'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":329,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:42.646126+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:51:42.646126+0000","last_peered":"2023-10-04T01:51:42.646126+0000","last_clean":"2023-10-04T01:51:42.646126+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:51:42.646126+0000","last_undegraded":"2023-10-04T01:51:42.646126+0000","last_fullsized":"2023-10-04T01:51:42.646126+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28067278867,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2015558,"num_read_kb":7939040,"num_write":2014221,"num_write_kb":8056878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16500060160,"data_stored":36187156518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58415,"ondisk_log_size":58415,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9542628,"kb_used_data":8057756,"kb_used_omap":2638,"kb_used_meta":1482225,"kb_avail":84829212,"statfs":{"total":96636764160,"available":86865113088,"internally_reserved":0,"allocated":8251142144,"data_stored":18093392885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":1517799030},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.51500000000000001,"15min":0.46200000000000002},"min":{"1min":0.183,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.52300000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":6.7930000000000001,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Wed Oct 4 01:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.921,"5min":0.91600000000000004,"15min":0.57999999999999996},"min":{"1min":0.28299999999999997,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":33.975000000000001,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.998,"5min":0.78600000000000003,"15min":0.57099999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":35.423999999999999,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.27000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739315,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6015032,"kb_used_data":3965936,"kb_used_omap":999,"kb_used_meta":2048088,"kb_avail":88356808,"statfs":{"total":96636764160,"available":90477371392,"internally_reserved":0,"allocated":4061118464,"data_stored":8910866821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1023220,"internal_metadata":2097242892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,3,9,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32300000000000001,"15min":0.34999999999999998},"min":{"1min":0.20999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41599999999999998,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56699999999999995,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.7169999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.29999999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.505,"15min":0.61399999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":1.978,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.58099999999999996,"15min":0.55600000000000005},"min":{"1min":0.23999999999999999,"5min":0.23499999999999999,"15min":0.20599999999999999},"max":{"1min":0.46500000000000002,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359739315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6047592,"kb_used_data":4095388,"kb_used_omap":1357,"kb_used_meta":1950834,"kb_avail":88324248,"statfs":{"total":96636764160,"available":90444029952,"internally_reserved":0,"allocated":4193677312,"data_stored":9184230620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1997654908},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,1,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:51:32 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36299999999999999,"15min":1.2390000000000001},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":0.69199999999999995,"5min":1.8600000000000001,"15min":207.047},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":1.3560000000000001},"min":{"1min":0.26100000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.859,"5min":5.6210000000000004,"15min":207.00200000000001},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39300000000000002,"15min":0.57199999999999995},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":0.52300000000000002,"5min":1.915,"15min":14.118},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42499999999999999,"15min":0.498},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":1.8,"5min":3.0720000000000001,"15min":14.051},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4191780864,"data_stored":9183756288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4058468352,"data_stored":8910041107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8249810944,"data_stored":18093359123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:52:11.796 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:11.796 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:11.797 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:11.797 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:11.797 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:52:23.269 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.257+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:52:23.276 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.269+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:52:23.276 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.269+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:52:23.276 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.269+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:52:23.276 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.269+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:52:23.276 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.269+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:52:23.277 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.273+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:52:23.277 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.273+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:52:23.278 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:52:23.273+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:52:31.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:52:32.323 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:52:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:52:32.353 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2530,"stamp":"2023-10-04T01:52:31.749779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28101640761,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2235259,"num_read_kb":8149845,"num_write":2078077,"num_write_kb":8145847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98742,"ondisk_log_size":98742,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21812036,"kb_used_data":16188140,"kb_used_omap":5187,"kb_used_meta":5618684,"kb_avail":261303484,"statfs":{"total":289910292480,"available":267574767616,"internally_reserved":0,"allocated":16576655360,"data_stored":36259207766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":5753533291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,0,5,11,19,12,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18944000,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4781,"num_read_kb":19048,"num_write":4601,"num_write_kb":18404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002099"},"pg_stats":[{"pgid":"2.7","version":"152'8493","reported_seq":11972,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:27.820919+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:52:27.820919+0000","last_peered":"2023-10-04T01:52:27.820919+0000","last_clean":"2023-10-04T01:52:27.820919+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:52:27.820919+0000","last_undegraded":"2023-10-04T01:52:27.820919+0000","last_fullsized":"2023-10-04T01:52:27.820919+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8493,"log_dups_size":0,"ondisk_log_size":8493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'7499","reported_seq":11242,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:24.462987+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:52:24.462987+0000","last_peered":"2023-10-04T01:52:24.462987+0000","last_clean":"2023-10-04T01:52:24.462987+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:52:24.462987+0000","last_undegraded":"2023-10-04T01:52:24.462987+0000","last_fullsized":"2023-10-04T01:52:24.462987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'8637","reported_seq":12423,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:27.816509+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:52:27.816509+0000","last_peered":"2023-10-04T01:52:27.816509+0000","last_clean":"2023-10-04T01:52:27.816509+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:52:27.816509+0000","last_undegraded":"2023-10-04T01:52:27.816509+0000","last_fullsized":"2023-10-04T01:52:27.816509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'7066","reported_seq":11698,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:27.821155+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:52:27.821155+0000","last_peered":"2023-10-04T01:52:27.821155+0000","last_clean":"2023-10-04T01:52:27.821155+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:52:27.821155+0000","last_undegraded":"2023-10-04T01:52:27.821155+0000","last_fullsized":"2023-10-04T01:52:27.821155+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7066,"log_dups_size":0,"ondisk_log_size":7066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'519614","reported_seq":1023402,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:31.018021+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:52:31.018021+0000","last_peered":"2023-10-04T01:52:31.018021+0000","last_clean":"2023-10-04T01:52:31.018021+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:52:31.018021+0000","last_undegraded":"2023-10-04T01:52:31.018021+0000","last_fullsized":"2023-10-04T01:52:31.018021+0000","mapping_epoch":15,"log_start":"150'509573","ondisk_log_start":"150'509573","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5679,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 806s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7056982016,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":508727,"num_read_kb":2003684,"num_write":508389,"num_write_kb":2033556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'17424","reported_seq":22742,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:29.578809+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:52:29.578809+0000","last_peered":"2023-10-04T01:52:29.578809+0000","last_clean":"2023-10-04T01:52:29.578809+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:52:29.578809+0000","last_undegraded":"2023-10-04T01:52:29.578809+0000","last_fullsized":"2023-10-04T01:52:29.578809+0000","mapping_epoch":12,"log_start":"85'7400","ondisk_log_start":"85'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'508421","reported_seq":1001139,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:27.903217+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:52:27.903217+0000","last_peered":"2023-10-04T01:52:27.903217+0000","last_clean":"2023-10-04T01:52:27.903217+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:52:27.903217+0000","last_undegraded":"2023-10-04T01:52:27.903217+0000","last_fullsized":"2023-10-04T01:52:27.903217+0000","mapping_epoch":15,"log_start":"150'498367","ondisk_log_start":"150'498367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5862,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 800s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6909243392,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":497557,"num_read_kb":1959768,"num_write":497145,"num_write_kb":1988580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9238,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:29.581838+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:52:29.581838+0000","last_peered":"2023-10-04T01:52:29.581838+0000","last_clean":"2023-10-04T01:52:29.581838+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:52:29.581838+0000","last_undegraded":"2023-10-04T01:52:29.581838+0000","last_fullsized":"2023-10-04T01:52:29.581838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'530453","reported_seq":1045703,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:31.012387+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:52:31.012387+0000","last_peered":"2023-10-04T01:52:31.012387+0000","last_clean":"2023-10-04T01:52:31.012387+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:52:31.012387+0000","last_undegraded":"2023-10-04T01:52:31.012387+0000","last_fullsized":"2023-10-04T01:52:31.012387+0000","mapping_epoch":15,"log_start":"150'520366","ondisk_log_start":"150'520366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5068,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 802s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7228858368,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":520360,"num_read_kb":2050340,"num_write":519565,"num_write_kb":2078260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9731,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:29.580364+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:52:29.580364+0000","last_peered":"2023-10-04T01:52:29.580364+0000","last_clean":"2023-10-04T01:52:29.580364+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:52:29.580364+0000","last_undegraded":"2023-10-04T01:52:29.580364+0000","last_fullsized":"2023-10-04T01:52:29.580364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'503846","reported_seq":996234,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:27.889363+0000","last_change":"2023-10-04T01:52:26.382881+0000","last_active":"2023-10-04T01:52:27.889363+0000","last_peered":"2023-10-04T01:52:27.889363+0000","last_clean":"2023-10-04T01:52:27.889363+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:52:27.889363+0000","last_undegraded":"2023-10-04T01:52:27.889363+0000","last_fullsized":"2023-10-04T01:52:27.889363+0000","mapping_epoch":15,"log_start":"150'493763","ondisk_log_start":"150'493763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6905950227,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":497441,"num_read_kb":1959200,"num_write":497334,"num_write_kb":1989330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'18328","reported_seq":23868,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:27.878985+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:52:27.878985+0000","last_peered":"2023-10-04T01:52:27.878985+0000","last_clean":"2023-10-04T01:52:27.878985+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:52:27.878985+0000","last_undegraded":"2023-10-04T01:52:27.878985+0000","last_fullsized":"2023-10-04T01:52:27.878985+0000","mapping_epoch":12,"log_start":"89'8300","ondisk_log_start":"89'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":329,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:42.646126+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:51:42.646126+0000","last_peered":"2023-10-04T01:51:42.646126+0000","last_clean":"2023-10-04T01:51:42.646126+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:51:42.646126+0000","last_undegraded":"2023-10-04T01:51:42.646126+0000","last_fullsized":"2023-10-04T01:51:42.646126+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28101034003,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2024085,"num_read_kb":7972992,"num_write":2022433,"num_write_kb":8089726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16570920960,"data_stored":36258017318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40265,"ondisk_log_size":40265,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58378,"ondisk_log_size":58378,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739322,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9709272,"kb_used_data":8093904,"kb_used_omap":2638,"kb_used_meta":1612721,"kb_avail":84662568,"statfs":{"total":96636764160,"available":86694469632,"internally_reserved":0,"allocated":8288157696,"data_stored":18130408437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":1651426934},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.51500000000000001,"15min":0.46200000000000002},"min":{"1min":0.183,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.52300000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":6.7930000000000001,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:51:47 2023","interfaces":[{"interface":"back","average":{"1min":1.921,"5min":0.91600000000000004,"15min":0.57999999999999996},"min":{"1min":0.28299999999999997,"5min":0.22700000000000001,"15min":0.20799999999999999},"max":{"1min":33.975000000000001,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.998,"5min":0.78600000000000003,"15min":0.57099999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":35.423999999999999,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.90900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739319,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5957640,"kb_used_data":3980480,"kb_used_omap":1191,"kb_used_meta":1975960,"kb_avail":88414200,"statfs":{"total":96636764160,"available":90536140800,"internally_reserved":0,"allocated":4076011520,"data_stored":8925759877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2023383417},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32300000000000001,"15min":0.34999999999999998},"min":{"1min":0.20999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41599999999999998,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.22},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56699999999999995,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.7169999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.505,"15min":0.61399999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":1.978,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.38},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.58099999999999996,"15min":0.55600000000000005},"min":{"1min":0.23999999999999999,"5min":0.23499999999999999,"15min":0.20599999999999999},"max":{"1min":0.46500000000000002,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6145124,"kb_used_data":4113756,"kb_used_omap":1357,"kb_used_meta":2030002,"kb_avail":88226716,"statfs":{"total":96636764160,"available":90344157184,"internally_reserved":0,"allocated":4212486144,"data_stored":9203039452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":2078722940},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,2,9,11,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:51:32 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.36299999999999999,"15min":1.2390000000000001},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":0.69199999999999995,"5min":1.8600000000000001,"15min":207.047},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":1.3560000000000001},"min":{"1min":0.26100000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.859,"5min":5.6210000000000004,"15min":207.00200000000001},"last":0.96699999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.39300000000000002,"15min":0.57199999999999995},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.22500000000000001},"max":{"1min":0.52300000000000002,"5min":1.915,"15min":14.118},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42499999999999999,"15min":0.498},"min":{"1min":0.24299999999999999,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":1.8,"5min":3.0720000000000001,"15min":14.051},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4210499584,"data_stored":9202475008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4073385984,"data_stored":8924958739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8287035392,"data_stored":18130583571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:52:32.357 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:32.357 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:32.357 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:32.357 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:32.358 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:52:51.310 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:52:51.304+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:52:52.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:52:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:52:52.885 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:52:52.913 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2540,"stamp":"2023-10-04T01:52:51.753084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28132168249,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2242740,"num_read_kb":8179637,"num_write":2085517,"num_write_kb":8175607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98825,"ondisk_log_size":98825,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21060948,"kb_used_data":16247740,"kb_used_omap":5187,"kb_used_meta":4807996,"kb_avail":262054572,"statfs":{"total":289910292480,"available":268343881728,"internally_reserved":0,"allocated":16637685760,"data_stored":36320238166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":4923388779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,6,13,10,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18743296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4644,"num_read_kb":18512,"num_write":4575,"num_write_kb":18300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001963"},"pg_stats":[{"pgid":"2.7","version":"152'8523","reported_seq":12010,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:47.931258+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:52:47.931258+0000","last_peered":"2023-10-04T01:52:47.931258+0000","last_clean":"2023-10-04T01:52:47.931258+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:52:47.931258+0000","last_undegraded":"2023-10-04T01:52:47.931258+0000","last_fullsized":"2023-10-04T01:52:47.931258+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'7514","reported_seq":11261,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:44.942932+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:52:44.942932+0000","last_peered":"2023-10-04T01:52:44.942932+0000","last_clean":"2023-10-04T01:52:44.942932+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:52:44.942932+0000","last_undegraded":"2023-10-04T01:52:44.942932+0000","last_fullsized":"2023-10-04T01:52:44.942932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'8667","reported_seq":12461,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:47.931189+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:52:47.931189+0000","last_peered":"2023-10-04T01:52:47.931189+0000","last_clean":"2023-10-04T01:52:47.931189+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:52:47.931189+0000","last_undegraded":"2023-10-04T01:52:47.931189+0000","last_fullsized":"2023-10-04T01:52:47.931189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'7096","reported_seq":11736,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:47.919058+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:52:47.919058+0000","last_peered":"2023-10-04T01:52:47.919058+0000","last_clean":"2023-10-04T01:52:47.919058+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:52:47.919058+0000","last_undegraded":"2023-10-04T01:52:47.919058+0000","last_fullsized":"2023-10-04T01:52:47.919058+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'521581","reported_seq":1027288,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:51.134247+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:52:51.134247+0000","last_peered":"2023-10-04T01:52:51.134247+0000","last_clean":"2023-10-04T01:52:51.134247+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:52:51.134247+0000","last_undegraded":"2023-10-04T01:52:51.134247+0000","last_fullsized":"2023-10-04T01:52:51.134247+0000","mapping_epoch":15,"log_start":"151'511573","ondisk_log_start":"151'511573","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5794,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 826s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7064674304,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":510636,"num_read_kb":2011300,"num_write":510262,"num_write_kb":2041048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"152'17466","reported_seq":22796,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:50.995369+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:52:50.995369+0000","last_peered":"2023-10-04T01:52:50.995369+0000","last_clean":"2023-10-04T01:52:50.995369+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:52:50.995369+0000","last_undegraded":"2023-10-04T01:52:50.995369+0000","last_fullsized":"2023-10-04T01:52:50.995369+0000","mapping_epoch":12,"log_start":"85'7400","ondisk_log_start":"85'7400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'510406","reported_seq":1004967,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:48.005947+0000","last_change":"2023-10-04T01:39:07.309330+0000","last_active":"2023-10-04T01:52:48.005947+0000","last_peered":"2023-10-04T01:52:48.005947+0000","last_clean":"2023-10-04T01:52:48.005947+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:52:48.005947+0000","last_undegraded":"2023-10-04T01:52:48.005947+0000","last_fullsized":"2023-10-04T01:52:48.005947+0000","mapping_epoch":15,"log_start":"151'500367","ondisk_log_start":"151'500367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5956,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 820s","scrub_duration":0,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6917095424,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":499392,"num_read_kb":1967084,"num_write":499052,"num_write_kb":1996208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9255,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:50.126214+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:52:50.126214+0000","last_peered":"2023-10-04T01:52:50.126214+0000","last_clean":"2023-10-04T01:52:50.126214+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:52:50.126214+0000","last_undegraded":"2023-10-04T01:52:50.126214+0000","last_fullsized":"2023-10-04T01:52:50.126214+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'532381","reported_seq":1049509,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:52:51.112301+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:52:51.112301+0000","last_peered":"2023-10-04T01:52:51.112301+0000","last_clean":"2023-10-04T01:52:51.112301+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:52:51.112301+0000","last_undegraded":"2023-10-04T01:52:51.112301+0000","last_fullsized":"2023-10-04T01:52:51.112301+0000","mapping_epoch":15,"log_start":"151'522366","ondisk_log_start":"151'522366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5164,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 822s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7236448256,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":522229,"num_read_kb":2057764,"num_write":521414,"num_write_kb":2085656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9743,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:50.141717+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:52:50.141717+0000","last_peered":"2023-10-04T01:52:50.141717+0000","last_clean":"2023-10-04T01:52:50.141717+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:52:50.141717+0000","last_undegraded":"2023-10-04T01:52:50.141717+0000","last_fullsized":"2023-10-04T01:52:50.141717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'505657","reported_seq":999928,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:47.987916+0000","last_change":"2023-10-04T01:52:45.315259+0000","last_active":"2023-10-04T01:52:47.987916+0000","last_peered":"2023-10-04T01:52:47.987916+0000","last_clean":"2023-10-04T01:52:47.987916+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:52:47.987916+0000","last_undegraded":"2023-10-04T01:52:47.987916+0000","last_fullsized":"2023-10-04T01:52:47.987916+0000","mapping_epoch":15,"log_start":"151'495563","ondisk_log_start":"151'495563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6913343507,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":499309,"num_read_kb":1966636,"num_write":499145,"num_write_kb":1996574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'18373","reported_seq":23925,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:47.961338+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:52:47.961338+0000","last_peered":"2023-10-04T01:52:47.961338+0000","last_clean":"2023-10-04T01:52:47.961338+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:52:47.961338+0000","last_undegraded":"2023-10-04T01:52:47.961338+0000","last_fullsized":"2023-10-04T01:52:47.961338+0000","mapping_epoch":12,"log_start":"89'8300","ondisk_log_start":"89'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":329,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:42.646126+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:51:42.646126+0000","last_peered":"2023-10-04T01:51:42.646126+0000","last_clean":"2023-10-04T01:51:42.646126+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:51:42.646126+0000","last_undegraded":"2023-10-04T01:51:42.646126+0000","last_fullsized":"2023-10-04T01:51:42.646126+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28131561491,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2031566,"num_read_kb":8002784,"num_write":2029873,"num_write_kb":8119486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16631742464,"data_stored":36318838822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40156,"ondisk_log_size":40156,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58570,"ondisk_log_size":58570,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739326,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9816704,"kb_used_data":8123256,"kb_used_omap":2638,"kb_used_meta":1690801,"kb_avail":84555136,"statfs":{"total":96636764160,"available":86584459264,"internally_reserved":0,"allocated":8318214144,"data_stored":18160464885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":1731380854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.51500000000000001,"15min":0.46200000000000002},"min":{"1min":0.183,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.52300000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":6.7930000000000001,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Wed Oct 4 01:52:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.82399999999999995,"15min":0.58199999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.73399999999999999,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.57199999999999995},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":5.2240000000000002,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739323,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6015052,"kb_used_data":3995268,"kb_used_omap":1191,"kb_used_meta":2018584,"kb_avail":88356788,"statfs":{"total":96636764160,"available":90477350912,"internally_reserved":0,"allocated":4091154432,"data_stored":8940902789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2067030393},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,0,4,9,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.32300000000000001,"15min":0.34999999999999998},"min":{"1min":0.20999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41599999999999998,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56699999999999995,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.7169999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.505,"15min":0.61399999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":1.978,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.252},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.58099999999999996,"15min":0.55600000000000005},"min":{"1min":0.23999999999999999,"5min":0.23499999999999999,"15min":0.20599999999999999},"max":{"1min":0.46500000000000002,"5min":6.1050000000000004,"15min":10.531000000000001},"last":0.29499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5229192,"kb_used_data":4129216,"kb_used_omap":1357,"kb_used_meta":1098610,"kb_avail":89142648,"statfs":{"total":96636764160,"available":91282071552,"internally_reserved":0,"allocated":4228317184,"data_stored":9218870492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1124977532},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.47999999999999998},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.0189999999999999,"5min":1.8600000000000001,"15min":17.991},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50600000000000001,"15min":0.59499999999999997},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.96699999999999997,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39800000000000002,"15min":0.57399999999999995},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.22500000000000001},"max":{"1min":0.99099999999999999,"5min":1.915,"15min":14.118},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.39400000000000002,"15min":0.498},"min":{"1min":0.223,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":0.56899999999999995,"5min":3.0720000000000001,"15min":14.051},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4225765376,"data_stored":9217740800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4088614912,"data_stored":8940187667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8317362176,"data_stored":18160910355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:52:52.917 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:52.917 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:52.917 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:52.918 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:52:52.918 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:53:12.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:53:13.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:53:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:53:13.481 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2551,"stamp":"2023-10-04T01:53:11.756518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28161319481,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2250008,"num_read_kb":8208561,"num_write":2092641,"num_write_kb":8204103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98826,"ondisk_log_size":98826,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21263300,"kb_used_data":16304684,"kb_used_omap":5187,"kb_used_meta":4953404,"kb_avail":261852220,"statfs":{"total":289910292480,"available":268136673280,"internally_reserved":0,"allocated":16695996416,"data_stored":36378540880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":5072286571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,10,6,15,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14618624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3516,"num_read_kb":13984,"num_write":3569,"num_write_kb":14276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001709"},"pg_stats":[{"pgid":"2.7","version":"152'8549","reported_seq":12044,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:07.848194+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:53:07.848194+0000","last_peered":"2023-10-04T01:53:07.848194+0000","last_clean":"2023-10-04T01:53:07.848194+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:53:07.848194+0000","last_undegraded":"2023-10-04T01:53:07.848194+0000","last_fullsized":"2023-10-04T01:53:07.848194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8549,"log_dups_size":0,"ondisk_log_size":8549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'7527","reported_seq":11278,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:07.490722+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:53:07.490722+0000","last_peered":"2023-10-04T01:53:07.490722+0000","last_clean":"2023-10-04T01:53:07.490722+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:53:07.490722+0000","last_undegraded":"2023-10-04T01:53:07.490722+0000","last_fullsized":"2023-10-04T01:53:07.490722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7527,"log_dups_size":0,"ondisk_log_size":7527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'8693","reported_seq":12496,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:07.839093+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:53:07.839093+0000","last_peered":"2023-10-04T01:53:07.839093+0000","last_clean":"2023-10-04T01:53:07.839093+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:53:07.839093+0000","last_undegraded":"2023-10-04T01:53:07.839093+0000","last_fullsized":"2023-10-04T01:53:07.839093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8693,"log_dups_size":0,"ondisk_log_size":8693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'7122","reported_seq":11771,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:07.840250+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:53:07.840250+0000","last_peered":"2023-10-04T01:53:07.840250+0000","last_clean":"2023-10-04T01:53:07.840250+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:53:07.840250+0000","last_undegraded":"2023-10-04T01:53:07.840250+0000","last_fullsized":"2023-10-04T01:53:07.840250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'523475","reported_seq":1031081,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:11.304816+0000","last_change":"2023-10-04T01:39:04.194154+0000","last_active":"2023-10-04T01:53:11.304816+0000","last_peered":"2023-10-04T01:53:11.304816+0000","last_clean":"2023-10-04T01:53:11.304816+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:53:11.304816+0000","last_undegraded":"2023-10-04T01:53:11.304816+0000","last_fullsized":"2023-10-04T01:53:11.304816+0000","mapping_epoch":15,"log_start":"151'513473","ondisk_log_start":"151'513473","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5975,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 847s","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7071801344,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":512524,"num_read_kb":2018808,"num_write":512009,"num_write_kb":2048036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'17528","reported_seq":22872,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:10.674444+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:53:10.674444+0000","last_peered":"2023-10-04T01:53:10.674444+0000","last_clean":"2023-10-04T01:53:10.674444+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:53:10.674444+0000","last_undegraded":"2023-10-04T01:53:10.674444+0000","last_fullsized":"2023-10-04T01:53:10.674444+0000","mapping_epoch":12,"log_start":"88'7500","ondisk_log_start":"88'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'512224","reported_seq":1008511,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:08.082357+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:53:08.082357+0000","last_peered":"2023-10-04T01:53:08.082357+0000","last_clean":"2023-10-04T01:53:08.082357+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:53:08.082357+0000","last_undegraded":"2023-10-04T01:53:08.082357+0000","last_fullsized":"2023-10-04T01:53:08.082357+0000","mapping_epoch":15,"log_start":"151'502167","ondisk_log_start":"151'502167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6924435456,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":501111,"num_read_kb":1973920,"num_write":500844,"num_write_kb":2003376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9272,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:10.547817+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:53:10.547817+0000","last_peered":"2023-10-04T01:53:10.547817+0000","last_clean":"2023-10-04T01:53:10.547817+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:53:10.547817+0000","last_undegraded":"2023-10-04T01:53:10.547817+0000","last_fullsized":"2023-10-04T01:53:10.547817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'534358","reported_seq":1053412,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:11.274626+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:53:11.274626+0000","last_peered":"2023-10-04T01:53:11.274626+0000","last_clean":"2023-10-04T01:53:11.274626+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:53:11.274626+0000","last_undegraded":"2023-10-04T01:53:11.274626+0000","last_fullsized":"2023-10-04T01:53:11.274626+0000","mapping_epoch":15,"log_start":"151'524266","ondisk_log_start":"151'524266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5316,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 843s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7244050432,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":524141,"num_read_kb":2065380,"num_write":523265,"num_write_kb":2093060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9756,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:10.543264+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:53:10.543264+0000","last_peered":"2023-10-04T01:53:10.543264+0000","last_clean":"2023-10-04T01:53:10.543264+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:53:10.543264+0000","last_undegraded":"2023-10-04T01:53:10.543264+0000","last_fullsized":"2023-10-04T01:53:10.543264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'507477","reported_seq":1003510,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:08.110025+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:53:08.110025+0000","last_peered":"2023-10-04T01:53:08.110025+0000","last_clean":"2023-10-04T01:53:08.110025+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:53:08.110025+0000","last_undegraded":"2023-10-04T01:53:08.110025+0000","last_fullsized":"2023-10-04T01:53:08.110025+0000","mapping_epoch":15,"log_start":"151'497463","ondisk_log_start":"151'497463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":103,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6920425491,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":501058,"num_read_kb":1973600,"num_write":500879,"num_write_kb":2003510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'18412","reported_seq":23977,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:53:07.879638+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:53:07.879638+0000","last_peered":"2023-10-04T01:53:07.879638+0000","last_clean":"2023-10-04T01:53:07.879638+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:53:07.879638+0000","last_undegraded":"2023-10-04T01:53:07.879638+0000","last_fullsized":"2023-10-04T01:53:07.879638+0000","mapping_epoch":12,"log_start":"92'8400","ondisk_log_start":"92'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":329,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-04T01:51:42.646126+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:51:42.646126+0000","last_peered":"2023-10-04T01:51:42.646126+0000","last_clean":"2023-10-04T01:51:42.646126+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:51:42.646126+0000","last_undegraded":"2023-10-04T01:51:42.646126+0000","last_fullsized":"2023-10-04T01:51:42.646126+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28160712723,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2038834,"num_read_kb":8031708,"num_write":2036997,"num_write_kb":8147982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16690122752,"data_stored":36377219110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40165,"ondisk_log_size":40165,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58562,"ondisk_log_size":58562,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739330,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9923572,"kb_used_data":8151596,"kb_used_omap":2638,"kb_used_meta":1769329,"kb_avail":84448268,"statfs":{"total":96636764160,"available":86475026432,"internally_reserved":0,"allocated":8347234304,"data_stored":18189481074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":1811793526},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.23400000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.54200000000000004,"15min":0.52600000000000002},"min":{"1min":0.29699999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.78900000000000003,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.31900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:52:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.82399999999999995,"15min":0.58199999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.73399999999999999,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.57199999999999995},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":5.2240000000000002,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739327,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6083448,"kb_used_data":4009520,"kb_used_omap":1191,"kb_used_meta":2072728,"kb_avail":88288392,"statfs":{"total":96636764160,"available":90407313408,"internally_reserved":0,"allocated":4105748480,"data_stored":8955496837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2122473849},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56000000000000005,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.503},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":6.431,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":1.0109999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.59199999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.5310000000000001,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":3.5659999999999998,"5min":3.601,"15min":10.531000000000001},"last":0.25}]}]},{"osd":0,"up_from":8,"seq":34359739328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5256280,"kb_used_data":4143568,"kb_used_omap":1357,"kb_used_meta":1111346,"kb_avail":89115560,"statfs":{"total":96636764160,"available":91254333440,"internally_reserved":0,"allocated":4243013632,"data_stored":9233562969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1138019196},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.47999999999999998},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.0189999999999999,"5min":1.8600000000000001,"15min":17.991},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50600000000000001,"15min":0.59499999999999997},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.96699999999999997,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39800000000000002,"15min":0.57399999999999995},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.22500000000000001},"max":{"1min":0.99099999999999999,"5min":1.915,"15min":14.118},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.39400000000000002,"15min":0.498},"min":{"1min":0.223,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":0.56899999999999995,"5min":3.0720000000000001,"15min":14.051},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4240490496,"data_stored":9232465920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4103069696,"data_stored":8954642451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8346562560,"data_stored":18190110739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:53:13.483 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:13.483 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:13.483 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:13.483 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:53:33.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:53:33.980 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:53:33.981 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:53:34.001 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2562,"stamp":"2023-10-04T01:53:33.762647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28195168825,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2258303,"num_read_kb":8241545,"num_write":2100908,"num_write_kb":8237171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99004,"ondisk_log_size":99004,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20489716,"kb_used_data":16366492,"kb_used_omap":5187,"kb_used_meta":4118012,"kb_avail":262625804,"statfs":{"total":289910292480,"available":268928823296,"internally_reserved":0,"allocated":16759287808,"data_stored":36441828301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":4216845163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,7,9,12,15,26,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19156992,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4697,"num_read_kb":18684,"num_write":4685,"num_write_kb":18740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004309"},"pg_stats":[{"pgid":"2.7","version":"153'8581","reported_seq":12087,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:32.609048+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:53:32.609048+0000","last_peered":"2023-10-04T01:53:32.609048+0000","last_clean":"2023-10-04T01:53:32.609048+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:53:32.609048+0000","last_undegraded":"2023-10-04T01:53:32.609048+0000","last_fullsized":"2023-10-04T01:53:32.609048+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'7543","reported_seq":11300,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:32.198206+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:53:32.198206+0000","last_peered":"2023-10-04T01:53:32.198206+0000","last_clean":"2023-10-04T01:53:32.198206+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:53:32.198206+0000","last_undegraded":"2023-10-04T01:53:32.198206+0000","last_fullsized":"2023-10-04T01:53:32.198206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'8725","reported_seq":12539,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:32.601873+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:53:32.601873+0000","last_peered":"2023-10-04T01:53:32.601873+0000","last_clean":"2023-10-04T01:53:32.601873+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:53:32.601873+0000","last_undegraded":"2023-10-04T01:53:32.601873+0000","last_fullsized":"2023-10-04T01:53:32.601873+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8725,"log_dups_size":0,"ondisk_log_size":8725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'7154","reported_seq":11814,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:32.598441+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:53:32.598441+0000","last_peered":"2023-10-04T01:53:32.598441+0000","last_clean":"2023-10-04T01:53:32.598441+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:53:32.598441+0000","last_undegraded":"2023-10-04T01:53:32.598441+0000","last_fullsized":"2023-10-04T01:53:32.598441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'525392","reported_seq":1034828,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:31.465970+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:53:31.465970+0000","last_peered":"2023-10-04T01:53:31.465970+0000","last_clean":"2023-10-04T01:53:31.465970+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:53:31.465970+0000","last_undegraded":"2023-10-04T01:53:31.465970+0000","last_fullsized":"2023-10-04T01:53:31.465970+0000","mapping_epoch":15,"log_start":"151'515373","ondisk_log_start":"151'515373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7079292928,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":514339,"num_read_kb":2026012,"num_write":513843,"num_write_kb":2055372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'17558","reported_seq":22914,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:31.103914+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:53:31.103914+0000","last_peered":"2023-10-04T01:53:31.103914+0000","last_clean":"2023-10-04T01:53:31.103914+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:53:31.103914+0000","last_undegraded":"2023-10-04T01:53:31.103914+0000","last_fullsized":"2023-10-04T01:53:31.103914+0000","mapping_epoch":12,"log_start":"88'7500","ondisk_log_start":"88'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'514514","reported_seq":1013130,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:33.256602+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:53:33.256602+0000","last_peered":"2023-10-04T01:53:33.256602+0000","last_clean":"2023-10-04T01:53:33.256602+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:53:33.256602+0000","last_undegraded":"2023-10-04T01:53:33.256602+0000","last_fullsized":"2023-10-04T01:53:33.256602+0000","mapping_epoch":15,"log_start":"151'504467","ondisk_log_start":"151'504467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6933860352,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":503445,"num_read_kb":1983208,"num_write":503134,"num_write_kb":2012536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9288,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:31.024344+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:53:31.024344+0000","last_peered":"2023-10-04T01:53:31.024344+0000","last_clean":"2023-10-04T01:53:31.024344+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:53:31.024344+0000","last_undegraded":"2023-10-04T01:53:31.024344+0000","last_fullsized":"2023-10-04T01:53:31.024344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'536344","reported_seq":1057314,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:31.460417+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:53:31.460417+0000","last_peered":"2023-10-04T01:53:31.460417+0000","last_clean":"2023-10-04T01:53:31.460417+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:53:31.460417+0000","last_undegraded":"2023-10-04T01:53:31.460417+0000","last_fullsized":"2023-10-04T01:53:31.460417+0000","mapping_epoch":15,"log_start":"151'526266","ondisk_log_start":"151'526266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5423,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 863s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7251812352,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":526047,"num_read_kb":2072964,"num_write":525161,"num_write_kb":2100644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9768,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:31.023069+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:53:31.023069+0000","last_peered":"2023-10-04T01:53:31.023069+0000","last_clean":"2023-10-04T01:53:31.023069+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:53:31.023069+0000","last_undegraded":"2023-10-04T01:53:31.023069+0000","last_fullsized":"2023-10-04T01:53:31.023069+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'509872","reported_seq":1008162,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:33.241550+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:53:33.241550+0000","last_peered":"2023-10-04T01:53:33.241550+0000","last_clean":"2023-10-04T01:53:33.241550+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:53:33.241550+0000","last_undegraded":"2023-10-04T01:53:33.241550+0000","last_fullsized":"2023-10-04T01:53:33.241550+0000","mapping_epoch":15,"log_start":"151'499863","ondisk_log_start":"151'499863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":283,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6929596435,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":503298,"num_read_kb":1982508,"num_write":503126,"num_write_kb":2012498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'18460","reported_seq":24041,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:32.632247+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:53:32.632247+0000","last_peered":"2023-10-04T01:53:32.632247+0000","last_clean":"2023-10-04T01:53:32.632247+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:53:32.632247+0000","last_undegraded":"2023-10-04T01:53:32.632247+0000","last_fullsized":"2023-10-04T01:53:32.632247+0000","mapping_epoch":12,"log_start":"92'8400","ondisk_log_start":"92'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":330,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-04T01:52:03.379248+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:52:03.379248+0000","last_peered":"2023-10-04T01:52:03.379248+0000","last_clean":"2023-10-04T01:52:03.379248+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:52:03.379248+0000","last_undegraded":"2023-10-04T01:52:03.379248+0000","last_fullsized":"2023-10-04T01:52:03.379248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28194562067,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2047129,"num_read_kb":8064692,"num_write":2045264,"num_write_kb":8181050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16753807360,"data_stored":36440903718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40153,"ondisk_log_size":40153,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58752,"ondisk_log_size":58752,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739334,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10067084,"kb_used_data":8180740,"kb_used_omap":2638,"kb_used_meta":1883697,"kb_avail":84304756,"statfs":{"total":96636764160,"available":86328070144,"internally_reserved":0,"allocated":8377077760,"data_stored":18219324530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":1928906358},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.23400000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.54200000000000004,"15min":0.52600000000000002},"min":{"1min":0.29699999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.78900000000000003,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:52:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.82399999999999995,"15min":0.58199999999999996},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.73399999999999999,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.79300000000000004,"15min":0.57199999999999995},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":5.2240000000000002,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739332,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5079452,"kb_used_data":4027412,"kb_used_omap":1191,"kb_used_meta":1050840,"kb_avail":89292388,"statfs":{"total":96636764160,"available":91435405312,"internally_reserved":0,"allocated":4124069888,"data_stored":8973814274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1076060537},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,6,6,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56000000000000005,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.503},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":6.431,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.35499999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.59199999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.5310000000000001,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":3.5659999999999998,"5min":3.601,"15min":10.531000000000001},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5343180,"kb_used_data":4158340,"kb_used_omap":1357,"kb_used_meta":1183474,"kb_avail":89028660,"statfs":{"total":96636764160,"available":91165347840,"internally_reserved":0,"allocated":4258140160,"data_stored":9248689497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1211878268},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,0,3,8,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.374,"15min":0.47999999999999998},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.188},"max":{"1min":1.0189999999999999,"5min":1.8600000000000001,"15min":17.991},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.50600000000000001,"15min":0.59499999999999997},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.96699999999999997,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.253}]},{"osd":2,"last update":"Wed Oct 4 01:52:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39800000000000002,"15min":0.57399999999999995},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.22500000000000001},"max":{"1min":0.99099999999999999,"5min":1.915,"15min":14.118},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.39400000000000002,"15min":0.498},"min":{"1min":0.223,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":0.56899999999999995,"5min":3.0720000000000001,"15min":14.051},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4255760384,"data_stored":9247735808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4121649152,"data_stored":8973221907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8376397824,"data_stored":18219946003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:53:34.003 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:34.003 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:34.004 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:53:54.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:53:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:53:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:53:54.517 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2572,"stamp":"2023-10-04T01:53:53.767714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28225868345,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2265605,"num_read_kb":8270605,"num_write":2108407,"num_write_kb":8267167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99236,"ondisk_log_size":99236,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20714388,"kb_used_data":16426492,"kb_used_omap":5187,"kb_used_meta":4282684,"kb_avail":262401132,"statfs":{"total":289910292480,"available":268698759168,"internally_reserved":0,"allocated":16820727808,"data_stored":36503268301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":4385469291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,5,9,21,11,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19546112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4649,"num_read_kb":18488,"num_write":4771,"num_write_kb":19084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002155"},"pg_stats":[{"pgid":"2.7","version":"153'8605","reported_seq":12119,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.802828+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:53:51.802828+0000","last_peered":"2023-10-04T01:53:51.802828+0000","last_clean":"2023-10-04T01:53:51.802828+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:53:51.802828+0000","last_undegraded":"2023-10-04T01:53:51.802828+0000","last_fullsized":"2023-10-04T01:53:51.802828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'7555","reported_seq":11317,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.498852+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:53:51.498852+0000","last_peered":"2023-10-04T01:53:51.498852+0000","last_clean":"2023-10-04T01:53:51.498852+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:53:51.498852+0000","last_undegraded":"2023-10-04T01:53:51.498852+0000","last_fullsized":"2023-10-04T01:53:51.498852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7555,"log_dups_size":0,"ondisk_log_size":7555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'8749","reported_seq":12571,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.807033+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:53:51.807033+0000","last_peered":"2023-10-04T01:53:51.807033+0000","last_clean":"2023-10-04T01:53:51.807033+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:53:51.807033+0000","last_undegraded":"2023-10-04T01:53:51.807033+0000","last_fullsized":"2023-10-04T01:53:51.807033+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'7178","reported_seq":11846,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.802359+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:53:51.802359+0000","last_peered":"2023-10-04T01:53:51.802359+0000","last_clean":"2023-10-04T01:53:51.802359+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:53:51.802359+0000","last_undegraded":"2023-10-04T01:53:51.802359+0000","last_fullsized":"2023-10-04T01:53:51.802359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7178,"log_dups_size":0,"ondisk_log_size":7178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'527311","reported_seq":1038582,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.607785+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:53:51.607785+0000","last_peered":"2023-10-04T01:53:51.607785+0000","last_clean":"2023-10-04T01:53:51.607785+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:53:51.607785+0000","last_undegraded":"2023-10-04T01:53:51.607785+0000","last_fullsized":"2023-10-04T01:53:51.607785+0000","mapping_epoch":15,"log_start":"152'517273","ondisk_log_start":"152'517273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7087153152,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":516175,"num_read_kb":2033328,"num_write":515762,"num_write_kb":2063048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'17585","reported_seq":22951,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.498952+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:53:51.498952+0000","last_peered":"2023-10-04T01:53:51.498952+0000","last_clean":"2023-10-04T01:53:51.498952+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:53:51.498952+0000","last_undegraded":"2023-10-04T01:53:51.498952+0000","last_fullsized":"2023-10-04T01:53:51.498952+0000","mapping_epoch":12,"log_start":"88'7500","ondisk_log_start":"88'7500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'516366","reported_seq":1016777,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:53.342866+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:53:53.342866+0000","last_peered":"2023-10-04T01:53:53.342866+0000","last_clean":"2023-10-04T01:53:53.342866+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:53:53.342866+0000","last_undegraded":"2023-10-04T01:53:53.342866+0000","last_fullsized":"2023-10-04T01:53:53.342866+0000","mapping_epoch":15,"log_start":"152'506267","ondisk_log_start":"152'506267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6941429760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":505234,"num_read_kb":1990316,"num_write":504986,"num_write_kb":2019944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9304,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.502142+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:53:51.502142+0000","last_peered":"2023-10-04T01:53:51.502142+0000","last_clean":"2023-10-04T01:53:51.502142+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:53:51.502142+0000","last_undegraded":"2023-10-04T01:53:51.502142+0000","last_fullsized":"2023-10-04T01:53:51.502142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'538356","reported_seq":1061158,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:51.607500+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:53:51.607500+0000","last_peered":"2023-10-04T01:53:51.607500+0000","last_clean":"2023-10-04T01:53:51.607500+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:53:51.607500+0000","last_undegraded":"2023-10-04T01:53:51.607500+0000","last_fullsized":"2023-10-04T01:53:51.607500+0000","mapping_epoch":15,"log_start":"152'528266","ondisk_log_start":"152'528266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5630,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 883s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7259377664,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":527865,"num_read_kb":2080208,"num_write":527008,"num_write_kb":2108032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9781,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.499052+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:53:51.499052+0000","last_peered":"2023-10-04T01:53:51.499052+0000","last_clean":"2023-10-04T01:53:51.499052+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:53:51.499052+0000","last_undegraded":"2023-10-04T01:53:51.499052+0000","last_fullsized":"2023-10-04T01:53:51.499052+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'511874","reported_seq":1012032,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:53:53.356545+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:53:53.356545+0000","last_peered":"2023-10-04T01:53:53.356545+0000","last_clean":"2023-10-04T01:53:53.356545+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:53:53.356545+0000","last_undegraded":"2023-10-04T01:53:53.356545+0000","last_fullsized":"2023-10-04T01:53:53.356545+0000","mapping_epoch":15,"log_start":"152'501863","ondisk_log_start":"152'501863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":433,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6937301011,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":505157,"num_read_kb":1989900,"num_write":505007,"num_write_kb":2020022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'18496","reported_seq":24089,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:51.837639+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:53:51.837639+0000","last_peered":"2023-10-04T01:53:51.837639+0000","last_clean":"2023-10-04T01:53:51.837639+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:53:51.837639+0000","last_undegraded":"2023-10-04T01:53:51.837639+0000","last_fullsized":"2023-10-04T01:53:51.837639+0000","mapping_epoch":12,"log_start":"92'8400","ondisk_log_start":"92'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":331,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:08.401713+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:53:08.401713+0000","last_peered":"2023-10-04T01:53:08.401713+0000","last_clean":"2023-10-04T01:53:08.401713+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:53:08.401713+0000","last_undegraded":"2023-10-04T01:53:08.401713+0000","last_fullsized":"2023-10-04T01:53:08.401713+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28225261587,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2054431,"num_read_kb":8093752,"num_write":2052763,"num_write_kb":8211046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16815415296,"data_stored":36502511654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40238,"ondisk_log_size":40238,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58899,"ondisk_log_size":58899,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739338,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10183168,"kb_used_data":8210808,"kb_used_omap":2638,"kb_used_meta":1969713,"kb_avail":84188672,"statfs":{"total":96636764160,"available":86209200128,"internally_reserved":0,"allocated":8407867392,"data_stored":18250114162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":2016986742},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.23400000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.54200000000000004,"15min":0.52600000000000002},"min":{"1min":0.29699999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.78900000000000003,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Oct 4 01:53:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.77400000000000002,"15min":0.58199999999999996},"min":{"1min":0.22800000000000001,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.69999999999999996,"5min":33.975000000000001,"15min":33.975000000000001},"last":1.349},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.73999999999999999,"15min":0.56999999999999995},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.48699999999999999,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.254}]}]},{"osd":1,"up_from":8,"seq":34359739336,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5154956,"kb_used_data":4042308,"kb_used_omap":1191,"kb_used_meta":1111448,"kb_avail":89216884,"statfs":{"total":96636764160,"available":91358089216,"internally_reserved":0,"allocated":4139323392,"data_stored":8989067778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1138123129},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,11,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56000000000000005,"5min":3.1850000000000001,"15min":3.1869999999999998},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.503},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":6.431,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.59199999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.5310000000000001,"5min":5.9790000000000001,"15min":10.494999999999999},"last":0.371},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47799999999999998,"15min":0.54100000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":3.5659999999999998,"5min":3.601,"15min":10.531000000000001},"last":0.40400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5376264,"kb_used_data":4173376,"kb_used_omap":1357,"kb_used_meta":1201522,"kb_avail":88995576,"statfs":{"total":96636764160,"available":91131469824,"internally_reserved":0,"allocated":4273537024,"data_stored":9264086361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1230359420},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,8,4,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:53:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.41099999999999998,"15min":0.48899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":17.991},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53700000000000003,"15min":0.60999999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":4.1139999999999999,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:53:51 2023","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":0.72499999999999998,"15min":0.66500000000000004},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":26.321999999999999,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.249,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.163,"5min":1.8,"15min":14.051},"last":1.163}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4271185920,"data_stored":9263161344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4136935424,"data_stored":8988508179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8407293952,"data_stored":18250842131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:53:54.519 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:54.519 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:53:54.519 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:54:03.980 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:54:03.971+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:54:14.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:54:15.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:54:15.154 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:54:15.189 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2582,"stamp":"2023-10-04T01:54:13.771031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28256674361,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2273398,"num_read_kb":8301569,"num_write":2115934,"num_write_kb":8297275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99009,"ondisk_log_size":99009,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20956392,"kb_used_data":16488144,"kb_used_omap":5187,"kb_used_meta":4463036,"kb_avail":262159128,"statfs":{"total":289910292480,"available":268450947072,"internally_reserved":0,"allocated":16883859456,"data_stored":36566399949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":4570149739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,4,4,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19369984,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4718,"num_read_kb":18760,"num_write":4732,"num_write_kb":18928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001961"},"pg_stats":[{"pgid":"2.7","version":"153'8613","reported_seq":12136,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.979184+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:54:11.979184+0000","last_peered":"2023-10-04T01:54:11.979184+0000","last_clean":"2023-10-04T01:54:11.979184+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:54:11.979184+0000","last_undegraded":"2023-10-04T01:54:11.979184+0000","last_fullsized":"2023-10-04T01:54:11.979184+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8613,"log_dups_size":0,"ondisk_log_size":8613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'7559","reported_seq":11325,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:07.356947+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:54:07.356947+0000","last_peered":"2023-10-04T01:54:07.356947+0000","last_clean":"2023-10-04T01:54:07.356947+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:54:07.356947+0000","last_undegraded":"2023-10-04T01:54:07.356947+0000","last_fullsized":"2023-10-04T01:54:07.356947+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'8757","reported_seq":12587,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.978550+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:54:11.978550+0000","last_peered":"2023-10-04T01:54:11.978550+0000","last_clean":"2023-10-04T01:54:11.978550+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:54:11.978550+0000","last_undegraded":"2023-10-04T01:54:11.978550+0000","last_fullsized":"2023-10-04T01:54:11.978550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'7186","reported_seq":11862,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.978630+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:54:11.978630+0000","last_peered":"2023-10-04T01:54:11.978630+0000","last_clean":"2023-10-04T01:54:11.978630+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:54:11.978630+0000","last_undegraded":"2023-10-04T01:54:11.978630+0000","last_fullsized":"2023-10-04T01:54:11.978630+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7186,"log_dups_size":0,"ondisk_log_size":7186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'529174","reported_seq":1042460,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.739217+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:54:11.739217+0000","last_peered":"2023-10-04T01:54:11.739217+0000","last_clean":"2023-10-04T01:54:11.739217+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:54:11.739217+0000","last_undegraded":"2023-10-04T01:54:11.739217+0000","last_fullsized":"2023-10-04T01:54:11.739217+0000","mapping_epoch":15,"log_start":"152'519173","ondisk_log_start":"152'519173","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7094767616,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":518189,"num_read_kb":2041324,"num_write":517625,"num_write_kb":2070500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"153'17633","reported_seq":23008,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:10.260962+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:54:10.260962+0000","last_peered":"2023-10-04T01:54:10.260962+0000","last_clean":"2023-10-04T01:54:10.260962+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:54:10.260962+0000","last_undegraded":"2023-10-04T01:54:10.260962+0000","last_fullsized":"2023-10-04T01:54:10.260962+0000","mapping_epoch":12,"log_start":"91'7600","ondisk_log_start":"91'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'518220","reported_seq":1020498,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:13.451218+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:54:13.451218+0000","last_peered":"2023-10-04T01:54:13.451218+0000","last_clean":"2023-10-04T01:54:13.451218+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:54:13.451218+0000","last_undegraded":"2023-10-04T01:54:13.451218+0000","last_fullsized":"2023-10-04T01:54:13.451218+0000","mapping_epoch":15,"log_start":"152'508167","ondisk_log_start":"152'508167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6949027840,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":507106,"num_read_kb":1997752,"num_write":506840,"num_write_kb":2027360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9320,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.981408+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:54:11.981408+0000","last_peered":"2023-10-04T01:54:11.981408+0000","last_clean":"2023-10-04T01:54:11.981408+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:54:11.981408+0000","last_undegraded":"2023-10-04T01:54:11.981408+0000","last_fullsized":"2023-10-04T01:54:11.981408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'540407","reported_seq":1065257,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:11.748088+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:54:11.748088+0000","last_peered":"2023-10-04T01:54:11.748088+0000","last_clean":"2023-10-04T01:54:11.748088+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:54:11.748088+0000","last_undegraded":"2023-10-04T01:54:11.748088+0000","last_fullsized":"2023-10-04T01:54:11.748088+0000","mapping_epoch":15,"log_start":"152'530366","ondisk_log_start":"152'530366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5787,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 903s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7267258368,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":529902,"num_read_kb":2088296,"num_write":528930,"num_write_kb":2115720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9793,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.980735+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:54:11.980735+0000","last_peered":"2023-10-04T01:54:11.980735+0000","last_clean":"2023-10-04T01:54:11.980735+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:54:11.980735+0000","last_undegraded":"2023-10-04T01:54:11.980735+0000","last_fullsized":"2023-10-04T01:54:11.980735+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'513891","reported_seq":1015925,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:13.459765+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:54:13.459765+0000","last_peered":"2023-10-04T01:54:13.459765+0000","last_clean":"2023-10-04T01:54:13.459765+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:54:13.459765+0000","last_undegraded":"2023-10-04T01:54:13.459765+0000","last_fullsized":"2023-10-04T01:54:13.459765+0000","mapping_epoch":15,"log_start":"152'503863","ondisk_log_start":"152'503863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":594,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6945013779,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":507027,"num_read_kb":1997344,"num_write":506895,"num_write_kb":2027574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'18508","reported_seq":24113,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:54:11.978999+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:54:11.978999+0000","last_peered":"2023-10-04T01:54:11.978999+0000","last_clean":"2023-10-04T01:54:11.978999+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:54:11.978999+0000","last_undegraded":"2023-10-04T01:54:11.978999+0000","last_fullsized":"2023-10-04T01:54:11.978999+0000","mapping_epoch":12,"log_start":"92'8500","ondisk_log_start":"92'8500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":331,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:08.401713+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:53:08.401713+0000","last_peered":"2023-10-04T01:53:08.401713+0000","last_clean":"2023-10-04T01:53:08.401713+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:53:08.401713+0000","last_undegraded":"2023-10-04T01:53:08.401713+0000","last_fullsized":"2023-10-04T01:53:08.401713+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28256067603,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2062224,"num_read_kb":8124716,"num_write":2060290,"num_write_kb":8241154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16877215744,"data_stored":36564312102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40123,"ondisk_log_size":40123,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":58787,"ondisk_log_size":58787,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739342,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10304800,"kb_used_data":8242264,"kb_used_omap":2638,"kb_used_meta":2059889,"kb_avail":84067040,"statfs":{"total":96636764160,"available":86084648960,"internally_reserved":0,"allocated":8440078336,"data_stored":18282325106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":2109326966},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41799999999999998,"15min":0.46700000000000003},"min":{"1min":0.19700000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.58699999999999997,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.47499999999999998,"15min":0.52700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.61499999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.436}]},{"osd":1,"last update":"Wed Oct 4 01:53:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.77400000000000002,"15min":0.58199999999999996},"min":{"1min":0.22800000000000001,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.69999999999999996,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.73999999999999999,"15min":0.56999999999999995},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.48699999999999999,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739340,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5188012,"kb_used_data":4057316,"kb_used_omap":1191,"kb_used_meta":1129496,"kb_avail":89183828,"statfs":{"total":96636764160,"available":91324239872,"internally_reserved":0,"allocated":4154691584,"data_stored":9004435970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1156604281},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.308,"15min":0.34499999999999997},"min":{"1min":0.21299999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.51400000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":3.5880000000000001,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.318}]},{"osd":2,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.44800000000000001,"15min":0.58999999999999997},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.641,"5min":4.641,"15min":10.494999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.54600000000000004},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":1.5109999999999999,"5min":3.5659999999999998,"15min":10.531000000000001},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5463580,"kb_used_data":4188564,"kb_used_omap":1357,"kb_used_meta":1273650,"kb_avail":88908260,"statfs":{"total":96636764160,"available":91042058240,"internally_reserved":0,"allocated":4289089536,"data_stored":9279638873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1304218492},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:53:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.41099999999999998,"15min":0.48899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":17.991},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53700000000000003,"15min":0.60999999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":4.1139999999999999,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.252}]},{"osd":2,"last update":"Wed Oct 4 01:53:51 2023","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":0.72499999999999998,"15min":0.66500000000000004},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":26.321999999999999,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.315},{"interface":"front","average":{"1min":0.374,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.249,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.163,"5min":1.8,"15min":14.051},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4286705664,"data_stored":9278681088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4152258560,"data_stored":9003831315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8438251520,"data_stored":18281799699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:54:15.190 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:15.190 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:15.191 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:54:19.086 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:19.078+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:54:19.097 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:19.090+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:54:19.097 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:19.090+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:54:19.097 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:19.090+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:54:33.661 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:33.654+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:54:33.662 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:33.654+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:54:33.662 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:33.654+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:54:34.793 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:34.778+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:54:34.793 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:34.778+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:54:34.794 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:34.778+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:54:34.794 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:34.778+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:54:35.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:54:35.685 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:54:35.685 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:54:35.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2593,"stamp":"2023-10-04T01:54:33.774929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28287574585,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2280945,"num_read_kb":8331609,"num_write":2123479,"num_write_kb":8327455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99457,"ondisk_log_size":99457,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21263624,"kb_used_data":16548336,"kb_used_omap":5187,"kb_used_meta":4710076,"kb_avail":261851896,"statfs":{"total":289910292480,"available":268136341504,"internally_reserved":0,"allocated":16945496064,"data_stored":36628024515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5311637,"internal_metadata":4823118699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,3,6,11,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22990848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5673,"num_read_kb":22608,"num_write":5614,"num_write_kb":22456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002471"},"pg_stats":[{"pgid":"2.7","version":"154'8655","reported_seq":12187,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:32.781339+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:54:32.781339+0000","last_peered":"2023-10-04T01:54:32.781339+0000","last_clean":"2023-10-04T01:54:32.781339+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:54:32.781339+0000","last_undegraded":"2023-10-04T01:54:32.781339+0000","last_fullsized":"2023-10-04T01:54:32.781339+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8655,"log_dups_size":0,"ondisk_log_size":8655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'7581","reported_seq":11352,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:32.458879+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:54:32.458879+0000","last_peered":"2023-10-04T01:54:32.458879+0000","last_clean":"2023-10-04T01:54:32.458879+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:54:32.458879+0000","last_undegraded":"2023-10-04T01:54:32.458879+0000","last_fullsized":"2023-10-04T01:54:32.458879+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'8801","reported_seq":12640,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:32.776425+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:54:32.776425+0000","last_peered":"2023-10-04T01:54:32.776425+0000","last_clean":"2023-10-04T01:54:32.776425+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:54:32.776425+0000","last_undegraded":"2023-10-04T01:54:32.776425+0000","last_fullsized":"2023-10-04T01:54:32.776425+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8801,"log_dups_size":0,"ondisk_log_size":8801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'7230","reported_seq":11915,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:32.776461+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:54:32.776461+0000","last_peered":"2023-10-04T01:54:32.776461+0000","last_clean":"2023-10-04T01:54:32.776461+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:54:32.776461+0000","last_undegraded":"2023-10-04T01:54:32.776461+0000","last_fullsized":"2023-10-04T01:54:32.776461+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'531151","reported_seq":1046327,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:31.848380+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:54:31.848380+0000","last_peered":"2023-10-04T01:54:31.848380+0000","last_clean":"2023-10-04T01:54:31.848380+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:54:31.848380+0000","last_undegraded":"2023-10-04T01:54:31.848380+0000","last_fullsized":"2023-10-04T01:54:31.848380+0000","mapping_epoch":15,"log_start":"152'521073","ondisk_log_start":"152'521073","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7102857216,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":520080,"num_read_kb":2048864,"num_write":519602,"num_write_kb":2078408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'17669","reported_seq":23057,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:29.398557+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:54:29.398557+0000","last_peered":"2023-10-04T01:54:29.398557+0000","last_clean":"2023-10-04T01:54:29.398557+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:54:29.398557+0000","last_undegraded":"2023-10-04T01:54:29.398557+0000","last_fullsized":"2023-10-04T01:54:29.398557+0000","mapping_epoch":12,"log_start":"91'7600","ondisk_log_start":"91'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'520053","reported_seq":1024207,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:33.552220+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:54:33.552220+0000","last_peered":"2023-10-04T01:54:33.552220+0000","last_clean":"2023-10-04T01:54:33.552220+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:54:33.552220+0000","last_undegraded":"2023-10-04T01:54:33.552220+0000","last_fullsized":"2023-10-04T01:54:33.552220+0000","mapping_epoch":15,"log_start":"152'509967","ondisk_log_start":"152'509967","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6956511232,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":508976,"num_read_kb":2005204,"num_write":508673,"num_write_kb":2034692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9333,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:27.342946+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:54:27.342946+0000","last_peered":"2023-10-04T01:54:27.342946+0000","last_clean":"2023-10-04T01:54:27.342946+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:54:27.342946+0000","last_undegraded":"2023-10-04T01:54:27.342946+0000","last_fullsized":"2023-10-04T01:54:27.342946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'542443","reported_seq":1069303,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:31.838772+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:54:31.838772+0000","last_peered":"2023-10-04T01:54:31.838772+0000","last_clean":"2023-10-04T01:54:31.838772+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:54:31.838772+0000","last_undegraded":"2023-10-04T01:54:31.838772+0000","last_fullsized":"2023-10-04T01:54:31.838772+0000","mapping_epoch":15,"log_start":"152'532366","ondisk_log_start":"152'532366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5935,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 923s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7275130880,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":531899,"num_read_kb":2096220,"num_write":530846,"num_write_kb":2123384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9803,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:27.362819+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:54:27.362819+0000","last_peered":"2023-10-04T01:54:27.362819+0000","last_clean":"2023-10-04T01:54:27.362819+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:54:27.362819+0000","last_undegraded":"2023-10-04T01:54:27.362819+0000","last_fullsized":"2023-10-04T01:54:27.362819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'515839","reported_seq":1019675,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:33.546188+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:54:33.546188+0000","last_peered":"2023-10-04T01:54:33.546188+0000","last_clean":"2023-10-04T01:54:33.546188+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:54:33.546188+0000","last_undegraded":"2023-10-04T01:54:33.546188+0000","last_fullsized":"2023-10-04T01:54:33.546188+0000","mapping_epoch":15,"log_start":"152'505763","ondisk_log_start":"152'505763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":750,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 98s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6952468499,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":508816,"num_read_kb":2004468,"num_write":508714,"num_write_kb":2034850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'18574","reported_seq":24192,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:32.798182+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:54:32.798182+0000","last_peered":"2023-10-04T01:54:32.798182+0000","last_clean":"2023-10-04T01:54:32.798182+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:54:32.798182+0000","last_undegraded":"2023-10-04T01:54:32.798182+0000","last_fullsized":"2023-10-04T01:54:32.798182+0000","mapping_epoch":12,"log_start":"92'8500","ondisk_log_start":"92'8500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":332,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:43.321931+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:53:43.321931+0000","last_peered":"2023-10-04T01:53:43.321931+0000","last_clean":"2023-10-04T01:53:43.321931+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:53:43.321931+0000","last_undegraded":"2023-10-04T01:53:43.321931+0000","last_fullsized":"2023-10-04T01:53:43.321931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28286967827,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2069771,"num_read_kb":8154756,"num_write":2067835,"num_write_kb":8271334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16938835968,"data_stored":36625932326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40317,"ondisk_log_size":40317,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64312,"internal_metadata":0},"log_size":59041,"ondisk_log_size":59041,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739346,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10473656,"kb_used_data":8272368,"kb_used_omap":2638,"kb_used_meta":2198641,"kb_avail":83898184,"statfs":{"total":96636764160,"available":85911740416,"internally_reserved":0,"allocated":8470904832,"data_stored":18313147588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2701706,"internal_metadata":2251409014},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41799999999999998,"15min":0.46700000000000003},"min":{"1min":0.19700000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.58699999999999997,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.47499999999999998,"15min":0.52700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.61499999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.622}]},{"osd":1,"last update":"Wed Oct 4 01:53:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.77400000000000002,"15min":0.58199999999999996},"min":{"1min":0.22800000000000001,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.69999999999999996,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.311},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.73999999999999999,"15min":0.56999999999999995},"min":{"1min":0.222,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.48699999999999999,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739344,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5274792,"kb_used_data":4071904,"kb_used_omap":1191,"kb_used_meta":1201688,"kb_avail":89097048,"statfs":{"total":96636764160,"available":91235377152,"internally_reserved":0,"allocated":4169629696,"data_stored":9019370068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1230528889},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.308,"15min":0.34499999999999997},"min":{"1min":0.21299999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.51400000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":3.5880000000000001,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.75600000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.44800000000000001,"15min":0.58999999999999997},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.641,"5min":4.641,"15min":10.494999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.54600000000000004},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":1.5109999999999999,"5min":3.5659999999999998,"15min":10.531000000000001},"last":0.435}]}]},{"osd":0,"up_from":8,"seq":34359739344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5515176,"kb_used_data":4204064,"kb_used_omap":1357,"kb_used_meta":1309746,"kb_avail":88856664,"statfs":{"total":96636764160,"available":90989223936,"internally_reserved":0,"allocated":4304961536,"data_stored":9295506859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1341180796},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:53:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.41099999999999998,"15min":0.48899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":17.991},"last":0.217},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53700000000000003,"15min":0.60999999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":4.1139999999999999,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.56999999999999995}]},{"osd":2,"last update":"Wed Oct 4 01:53:51 2023","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":0.72499999999999998,"15min":0.66500000000000004},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":26.321999999999999,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.374,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.249,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.163,"5min":1.8,"15min":14.051},"last":0.32900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19995,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4302635008,"data_stored":9294610432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4167213056,"data_stored":9018785811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8468987904,"data_stored":18312536083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:54:35.715 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:35.715 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:35.715 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:54:53.686 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.681+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:54:53.691 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:54:53.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:54:53.693 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:54:53.685+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:54:55.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:54:56.218 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:54:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:54:56.261 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2604,"stamp":"2023-10-04T01:54:55.778758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28318085689,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2288383,"num_read_kb":8361201,"num_write":2130941,"num_write_kb":8357303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99366,"ondisk_log_size":99366,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21086336,"kb_used_data":16607720,"kb_used_omap":5176,"kb_used_meta":4473415,"kb_avail":262029184,"statfs":{"total":289910292480,"available":268317884416,"internally_reserved":0,"allocated":17006305280,"data_stored":36688833731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4580777084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3,7,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15216640,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3664,"num_read_kb":14588,"num_write":3723,"num_write_kb":14892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002028"},"pg_stats":[{"pgid":"2.7","version":"154'8683","reported_seq":12223,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:52.938805+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:54:52.938805+0000","last_peered":"2023-10-04T01:54:52.938805+0000","last_clean":"2023-10-04T01:54:52.938805+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:54:52.938805+0000","last_undegraded":"2023-10-04T01:54:52.938805+0000","last_fullsized":"2023-10-04T01:54:52.938805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8683,"log_dups_size":0,"ondisk_log_size":8683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'7595","reported_seq":11370,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:52.434485+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:54:52.434485+0000","last_peered":"2023-10-04T01:54:52.434485+0000","last_clean":"2023-10-04T01:54:52.434485+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:54:52.434485+0000","last_undegraded":"2023-10-04T01:54:52.434485+0000","last_fullsized":"2023-10-04T01:54:52.434485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'8829","reported_seq":12676,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:52.938620+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:54:52.938620+0000","last_peered":"2023-10-04T01:54:52.938620+0000","last_clean":"2023-10-04T01:54:52.938620+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:54:52.938620+0000","last_undegraded":"2023-10-04T01:54:52.938620+0000","last_fullsized":"2023-10-04T01:54:52.938620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'7258","reported_seq":11951,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:52.938636+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:54:52.938636+0000","last_peered":"2023-10-04T01:54:52.938636+0000","last_clean":"2023-10-04T01:54:52.938636+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:54:52.938636+0000","last_undegraded":"2023-10-04T01:54:52.938636+0000","last_fullsized":"2023-10-04T01:54:52.938636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7258,"log_dups_size":0,"ondisk_log_size":7258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'532996","reported_seq":1050020,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:51.913313+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:54:51.913313+0000","last_peered":"2023-10-04T01:54:51.913313+0000","last_clean":"2023-10-04T01:54:51.913313+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:54:51.913313+0000","last_undegraded":"2023-10-04T01:54:51.913313+0000","last_fullsized":"2023-10-04T01:54:51.913313+0000","mapping_epoch":15,"log_start":"152'522973","ondisk_log_start":"152'522973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7110410240,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":521929,"num_read_kb":2056224,"num_write":521447,"num_write_kb":2085788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'17687","reported_seq":23087,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:51.272155+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:54:51.272155+0000","last_peered":"2023-10-04T01:54:51.272155+0000","last_clean":"2023-10-04T01:54:51.272155+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:54:51.272155+0000","last_undegraded":"2023-10-04T01:54:51.272155+0000","last_fullsized":"2023-10-04T01:54:51.272155+0000","mapping_epoch":12,"log_start":"91'7600","ondisk_log_start":"91'7600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'521873","reported_seq":1027834,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:53.663591+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:54:53.663591+0000","last_peered":"2023-10-04T01:54:53.663591+0000","last_clean":"2023-10-04T01:54:53.663591+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:54:53.663591+0000","last_undegraded":"2023-10-04T01:54:53.663591+0000","last_fullsized":"2023-10-04T01:54:53.663591+0000","mapping_epoch":15,"log_start":"152'511867","ondisk_log_start":"152'511867","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6963949568,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":510784,"num_read_kb":2012400,"num_write":510493,"num_write_kb":2041972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9350,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:47.824340+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:54:47.824340+0000","last_peered":"2023-10-04T01:54:47.824340+0000","last_clean":"2023-10-04T01:54:47.824340+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:54:47.824340+0000","last_undegraded":"2023-10-04T01:54:47.824340+0000","last_fullsized":"2023-10-04T01:54:47.824340+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'544537","reported_seq":1073323,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:51.917259+0000","last_change":"2023-10-04T01:39:08.131462+0000","last_active":"2023-10-04T01:54:51.917259+0000","last_peered":"2023-10-04T01:54:51.917259+0000","last_clean":"2023-10-04T01:54:51.917259+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:54:51.917259+0000","last_undegraded":"2023-10-04T01:54:51.917259+0000","last_fullsized":"2023-10-04T01:54:51.917259+0000","mapping_epoch":15,"log_start":"153'534466","ondisk_log_start":"153'534466","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":6125,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 943s","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7283023872,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":533810,"num_read_kb":2103824,"num_write":532781,"num_write_kb":2131124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9815,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:47.823515+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:54:47.823515+0000","last_peered":"2023-10-04T01:54:47.823515+0000","last_clean":"2023-10-04T01:54:47.823515+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:54:47.823515+0000","last_undegraded":"2023-10-04T01:54:47.823515+0000","last_fullsized":"2023-10-04T01:54:47.823515+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'517832","reported_seq":1023559,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:54:53.661050+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:54:53.661050+0000","last_peered":"2023-10-04T01:54:53.661050+0000","last_clean":"2023-10-04T01:54:53.661050+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:54:53.661050+0000","last_undegraded":"2023-10-04T01:54:53.661050+0000","last_fullsized":"2023-10-04T01:54:53.661050+0000","mapping_epoch":15,"log_start":"153'507763","ondisk_log_start":"153'507763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":916,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6960095251,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":510686,"num_read_kb":2011900,"num_write":510576,"num_write_kb":2042298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'18615","reported_seq":24245,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:52.939189+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:54:52.939189+0000","last_peered":"2023-10-04T01:54:52.939189+0000","last_clean":"2023-10-04T01:54:52.939189+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:54:52.939189+0000","last_undegraded":"2023-10-04T01:54:52.939189+0000","last_fullsized":"2023-10-04T01:54:52.939189+0000","mapping_epoch":12,"log_start":"95'8600","ondisk_log_start":"95'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":332,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:43.321931+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:53:43.321931+0000","last_peered":"2023-10-04T01:53:43.321931+0000","last_clean":"2023-10-04T01:53:43.321931+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:53:43.321931+0000","last_undegraded":"2023-10-04T01:53:43.321931+0000","last_fullsized":"2023-10-04T01:53:43.321931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28317478931,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2077209,"num_read_kb":8184348,"num_write":2075297,"num_write_kb":8301182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16999841792,"data_stored":36686938150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40169,"ondisk_log_size":40169,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59098,"ondisk_log_size":59098,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739350,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10194264,"kb_used_data":8301840,"kb_used_omap":2628,"kb_used_meta":1889787,"kb_avail":84177576,"statfs":{"total":96636764160,"available":86197837824,"internally_reserved":0,"allocated":8501084160,"data_stored":18343326916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1935142791},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41799999999999998,"15min":0.46700000000000003},"min":{"1min":0.19700000000000001,"5min":0.17599999999999999,"15min":0.16500000000000001},"max":{"1min":0.58699999999999997,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":4.4580000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.47499999999999998,"15min":0.52700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":0.61499999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":4.492}]},{"osd":1,"last update":"Wed Oct 4 01:54:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.749,"15min":0.58199999999999996},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.349,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.71999999999999997,"15min":0.56399999999999995},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.38800000000000001,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739348,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5325564,"kb_used_data":4086580,"kb_used_omap":1191,"kb_used_meta":1237784,"kb_avail":89046276,"statfs":{"total":96636764160,"available":91183386624,"internally_reserved":0,"allocated":4184657920,"data_stored":9034398292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1267491193},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.308,"15min":0.34499999999999997},"min":{"1min":0.21299999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.189},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.51400000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":3.5880000000000001,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":4.2699999999999996}]},{"osd":2,"last update":"Wed Oct 4 01:53:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.44800000000000001,"15min":0.58999999999999997},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.641,"5min":4.641,"15min":10.494999999999999},"last":0.221},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.54600000000000004},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":1.5109999999999999,"5min":3.5659999999999998,"15min":10.531000000000001},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5566508,"kb_used_data":4219300,"kb_used_omap":1357,"kb_used_meta":1345842,"kb_avail":88805332,"statfs":{"total":96636764160,"available":90936659968,"internally_reserved":0,"allocated":4320563200,"data_stored":9311108523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1378143100},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41899999999999998,"15min":0.48899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.2130000000000001,"5min":4.0730000000000004,"15min":17.991},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.52800000000000002,"15min":0.60999999999999999},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":0.76200000000000001,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:53:51 2023","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":0.72499999999999998,"15min":0.66500000000000004},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":26.321999999999999,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.36099999999999999,"15min":0.46500000000000002},"min":{"1min":0.249,"5min":0.223,"15min":0.20200000000000001},"max":{"1min":1.163,"5min":1.8,"15min":14.051},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4318121984,"data_stored":9310097408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4182298624,"data_stored":9033871379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8499421184,"data_stored":18342969363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:54:56.263 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:56.263 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:54:56.264 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:55:12.474 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.469+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:55:12.488 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:55:12.488 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:55:12.489 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:55:12.489 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:55:12.489 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:55:12.489 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:55:12.489 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:55:12.490 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:55:12.490 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:55:12.485+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:55:16.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:55:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:55:16.798 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:55:16.830 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2614,"stamp":"2023-10-04T01:55:15.782036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28349768249,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2296177,"num_read_kb":8392225,"num_write":2138674,"num_write_kb":8388235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99475,"ondisk_log_size":99475,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21324016,"kb_used_data":16669080,"kb_used_omap":5176,"kb_used_meta":4649735,"kb_avail":261791504,"statfs":{"total":289910292480,"available":268074500096,"internally_reserved":0,"allocated":17069137920,"data_stored":36751666371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4761328764},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,7,12,16],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16343040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4084,"num_read_kb":16256,"num_write":3994,"num_write_kb":15976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001996"},"pg_stats":[{"pgid":"2.7","version":"154'8709","reported_seq":12257,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.503990+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:55:13.503990+0000","last_peered":"2023-10-04T01:55:13.503990+0000","last_clean":"2023-10-04T01:55:13.503990+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:55:13.503990+0000","last_undegraded":"2023-10-04T01:55:13.503990+0000","last_fullsized":"2023-10-04T01:55:13.503990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'7608","reported_seq":11387,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.422826+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:55:13.422826+0000","last_peered":"2023-10-04T01:55:13.422826+0000","last_clean":"2023-10-04T01:55:13.422826+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:55:13.422826+0000","last_undegraded":"2023-10-04T01:55:13.422826+0000","last_fullsized":"2023-10-04T01:55:13.422826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'8855","reported_seq":12711,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.494723+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:55:13.494723+0000","last_peered":"2023-10-04T01:55:13.494723+0000","last_clean":"2023-10-04T01:55:13.494723+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:55:13.494723+0000","last_undegraded":"2023-10-04T01:55:13.494723+0000","last_fullsized":"2023-10-04T01:55:13.494723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'7284","reported_seq":11986,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.494602+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:55:13.494602+0000","last_peered":"2023-10-04T01:55:13.494602+0000","last_clean":"2023-10-04T01:55:13.494602+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:55:13.494602+0000","last_undegraded":"2023-10-04T01:55:13.494602+0000","last_fullsized":"2023-10-04T01:55:13.494602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'534892","reported_seq":1053926,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:12.000240+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:55:12.000240+0000","last_peered":"2023-10-04T01:55:12.000240+0000","last_clean":"2023-10-04T01:55:12.000240+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:55:12.000240+0000","last_undegraded":"2023-10-04T01:55:12.000240+0000","last_fullsized":"2023-10-04T01:55:12.000240+0000","mapping_epoch":15,"log_start":"153'524873","ondisk_log_start":"153'524873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7118188544,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":523938,"num_read_kb":2064228,"num_write":523343,"num_write_kb":2093372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"154'17732","reported_seq":23145,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:08.864477+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:55:08.864477+0000","last_peered":"2023-10-04T01:55:08.864477+0000","last_clean":"2023-10-04T01:55:08.864477+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:55:08.864477+0000","last_undegraded":"2023-10-04T01:55:08.864477+0000","last_fullsized":"2023-10-04T01:55:08.864477+0000","mapping_epoch":12,"log_start":"92'7700","ondisk_log_start":"92'7700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'523740","reported_seq":1031608,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.758859+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:55:13.758859+0000","last_peered":"2023-10-04T01:55:13.758859+0000","last_clean":"2023-10-04T01:55:13.758859+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:55:13.758859+0000","last_undegraded":"2023-10-04T01:55:13.758859+0000","last_fullsized":"2023-10-04T01:55:13.758859+0000","mapping_epoch":15,"log_start":"153'513667","ondisk_log_start":"153'513667","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6971592704,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":512691,"num_read_kb":2019972,"num_write":512360,"num_write_kb":2049440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9366,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:08.299522+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:55:08.299522+0000","last_peered":"2023-10-04T01:55:08.299522+0000","last_clean":"2023-10-04T01:55:08.299522+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:55:08.299522+0000","last_undegraded":"2023-10-04T01:55:08.299522+0000","last_fullsized":"2023-10-04T01:55:08.299522+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'546636","reported_seq":1077420,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:11.995682+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:55:11.995682+0000","last_peered":"2023-10-04T01:55:11.995682+0000","last_clean":"2023-10-04T01:55:11.995682+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:55:11.995682+0000","last_undegraded":"2023-10-04T01:55:11.995682+0000","last_fullsized":"2023-10-04T01:55:11.995682+0000","mapping_epoch":15,"log_start":"153'536566","ondisk_log_start":"153'536566","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7291203584,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":535796,"num_read_kb":2111732,"num_write":534775,"num_write_kb":2139100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9827,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:08.299050+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:55:08.299050+0000","last_peered":"2023-10-04T01:55:08.299050+0000","last_clean":"2023-10-04T01:55:08.299050+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:55:08.299050+0000","last_undegraded":"2023-10-04T01:55:08.299050+0000","last_fullsized":"2023-10-04T01:55:08.299050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'519904","reported_seq":1027536,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:55:13.777125+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:55:13.777125+0000","last_peered":"2023-10-04T01:55:13.777125+0000","last_clean":"2023-10-04T01:55:13.777125+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:55:13.777125+0000","last_undegraded":"2023-10-04T01:55:13.777125+0000","last_fullsized":"2023-10-04T01:55:13.777125+0000","mapping_epoch":15,"log_start":"153'509863","ondisk_log_start":"153'509863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1031,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6968176659,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":512578,"num_read_kb":2019440,"num_write":512552,"num_write_kb":2050202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'18654","reported_seq":24297,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:55:13.541831+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:55:13.541831+0000","last_peered":"2023-10-04T01:55:13.541831+0000","last_clean":"2023-10-04T01:55:13.541831+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:55:13.541831+0000","last_undegraded":"2023-10-04T01:55:13.541831+0000","last_fullsized":"2023-10-04T01:55:13.541831+0000","mapping_epoch":12,"log_start":"95'8600","ondisk_log_start":"95'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":332,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-04T01:53:43.321931+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:53:43.321931+0000","last_peered":"2023-10-04T01:53:43.321931+0000","last_clean":"2023-10-04T01:53:43.321931+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:53:43.321931+0000","last_undegraded":"2023-10-04T01:53:43.321931+0000","last_fullsized":"2023-10-04T01:53:43.321931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28349161491,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2085003,"num_read_kb":8215372,"num_write":2083030,"num_write_kb":8332114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17063170048,"data_stored":36750266406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40203,"ondisk_log_size":40203,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59173,"ondisk_log_size":59173,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739354,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10292724,"kb_used_data":8332268,"kb_used_omap":2628,"kb_used_meta":1957819,"kb_avail":84079116,"statfs":{"total":96636764160,"available":86097014784,"internally_reserved":0,"allocated":8532242432,"data_stored":18374485188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":2004807559},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:55:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":4.4580000000000002,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.492,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.26500000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:54:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.749,"15min":0.58199999999999996},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.349,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.71999999999999997,"15min":0.56399999999999995},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.38800000000000001,"5min":35.423999999999999,"15min":35.423999999999999},"last":3.706}]}]},{"osd":1,"up_from":8,"seq":34359739352,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5413112,"kb_used_data":4101936,"kb_used_omap":1191,"kb_used_meta":1309976,"kb_avail":88958728,"statfs":{"total":96636764160,"available":91093737472,"internally_reserved":0,"allocated":4200382464,"data_stored":9050122836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1341415801},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.313,"15min":0.33900000000000002},"min":{"1min":0.189,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.52800000000000002},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":4.2699999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.41299999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.51700000000000002,"15min":0.58699999999999997},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.96,"5min":4.641,"15min":10.494999999999999},"last":0.433},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.49299999999999999,"15min":0.56000000000000005},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.376}]}]},{"osd":0,"up_from":8,"seq":34359739352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5618180,"kb_used_data":4234876,"kb_used_omap":1357,"kb_used_meta":1381938,"kb_avail":88753660,"statfs":{"total":96636764160,"available":90883747840,"internally_reserved":0,"allocated":4336513024,"data_stored":9327058347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1415105404},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41899999999999998,"15min":0.48899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.2130000000000001,"5min":4.0730000000000004,"15min":17.991},"last":1.6859999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.52800000000000002,"15min":0.60999999999999999},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":0.76200000000000001,"5min":5.6210000000000004,"15min":17.957000000000001},"last":1.7569999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:54:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.73099999999999998,"15min":0.65800000000000003},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.56499999999999995,"5min":26.321999999999999,"15min":26.321999999999999},"last":1.72},{"interface":"front","average":{"1min":0.442,"5min":0.375,"15min":0.46600000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":2.597,"5min":2.597,"15min":14.051},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4334063616,"data_stored":9326039040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4198039552,"data_stored":9049612307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8531066880,"data_stored":18374615059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:55:16.831 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:55:16.832 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:55:36.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:55:37.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:55:37.333 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:55:37.368 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2625,"stamp":"2023-10-04T01:55:35.785611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28382200377,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2304197,"num_read_kb":8424157,"num_write":2146584,"num_write_kb":8419875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99662,"ondisk_log_size":99662,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21533336,"kb_used_data":16733120,"kb_used_omap":5176,"kb_used_meta":4795015,"kb_avail":261582184,"statfs":{"total":289910292480,"available":267860156416,"internally_reserved":0,"allocated":17134714880,"data_stored":36817231160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4910095484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,5,7,15,13,22,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16658432,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4130,"num_read_kb":16448,"num_write":4058,"num_write_kb":16232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001685"},"pg_stats":[{"pgid":"2.7","version":"155'8737","reported_seq":12292,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.282587+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:55:32.282587+0000","last_peered":"2023-10-04T01:55:32.282587+0000","last_clean":"2023-10-04T01:55:32.282587+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:55:32.282587+0000","last_undegraded":"2023-10-04T01:55:32.282587+0000","last_fullsized":"2023-10-04T01:55:32.282587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8737,"log_dups_size":0,"ondisk_log_size":8737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'7622","reported_seq":11405,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:31.471796+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:55:31.471796+0000","last_peered":"2023-10-04T01:55:31.471796+0000","last_clean":"2023-10-04T01:55:31.471796+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:55:31.471796+0000","last_undegraded":"2023-10-04T01:55:31.471796+0000","last_fullsized":"2023-10-04T01:55:31.471796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'8883","reported_seq":12746,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.282381+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:55:32.282381+0000","last_peered":"2023-10-04T01:55:32.282381+0000","last_clean":"2023-10-04T01:55:32.282381+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:55:32.282381+0000","last_undegraded":"2023-10-04T01:55:32.282381+0000","last_fullsized":"2023-10-04T01:55:32.282381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'7312","reported_seq":12021,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.276997+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:55:32.276997+0000","last_peered":"2023-10-04T01:55:32.276997+0000","last_clean":"2023-10-04T01:55:32.276997+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:55:32.276997+0000","last_undegraded":"2023-10-04T01:55:32.276997+0000","last_fullsized":"2023-10-04T01:55:32.276997+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7312,"log_dups_size":0,"ondisk_log_size":7312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'536942","reported_seq":1057939,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.079240+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:55:32.079240+0000","last_peered":"2023-10-04T01:55:32.079240+0000","last_clean":"2023-10-04T01:55:32.079240+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:55:32.079240+0000","last_undegraded":"2023-10-04T01:55:32.079240+0000","last_fullsized":"2023-10-04T01:55:32.079240+0000","mapping_epoch":15,"log_start":"153'526873","ondisk_log_start":"153'526873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7126605824,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":525896,"num_read_kb":2072040,"num_write":525393,"num_write_kb":2101572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'17775","reported_seq":23201,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:29.319232+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:55:29.319232+0000","last_peered":"2023-10-04T01:55:29.319232+0000","last_clean":"2023-10-04T01:55:29.319232+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:55:29.319232+0000","last_undegraded":"2023-10-04T01:55:29.319232+0000","last_fullsized":"2023-10-04T01:55:29.319232+0000","mapping_epoch":12,"log_start":"92'7700","ondisk_log_start":"92'7700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'525662","reported_seq":1035536,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:33.849483+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:55:33.849483+0000","last_peered":"2023-10-04T01:55:33.849483+0000","last_clean":"2023-10-04T01:55:33.849483+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:55:33.849483+0000","last_undegraded":"2023-10-04T01:55:33.849483+0000","last_fullsized":"2023-10-04T01:55:33.849483+0000","mapping_epoch":15,"log_start":"153'515567","ondisk_log_start":"153'515567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6979506176,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":514696,"num_read_kb":2027944,"num_write":514282,"num_write_kb":2057128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9383,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:28.788704+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:55:28.788704+0000","last_peered":"2023-10-04T01:55:28.788704+0000","last_clean":"2023-10-04T01:55:28.788704+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:55:28.788704+0000","last_undegraded":"2023-10-04T01:55:28.788704+0000","last_fullsized":"2023-10-04T01:55:28.788704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'548692","reported_seq":1081557,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.095409+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:55:32.095409+0000","last_peered":"2023-10-04T01:55:32.095409+0000","last_clean":"2023-10-04T01:55:32.095409+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:55:32.095409+0000","last_undegraded":"2023-10-04T01:55:32.095409+0000","last_fullsized":"2023-10-04T01:55:32.095409+0000","mapping_epoch":15,"log_start":"153'538666","ondisk_log_start":"153'538666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7299620864,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":537876,"num_read_kb":2120020,"num_write":536831,"num_write_kb":2147324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9840,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:28.783455+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:55:28.783455+0000","last_peered":"2023-10-04T01:55:28.783455+0000","last_clean":"2023-10-04T01:55:28.783455+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:55:28.783455+0000","last_undegraded":"2023-10-04T01:55:28.783455+0000","last_fullsized":"2023-10-04T01:55:28.783455+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'521880","reported_seq":1031498,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:55:33.868212+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:55:33.868212+0000","last_peered":"2023-10-04T01:55:33.868212+0000","last_clean":"2023-10-04T01:55:33.868212+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:55:33.868212+0000","last_undegraded":"2023-10-04T01:55:33.868212+0000","last_fullsized":"2023-10-04T01:55:33.868212+0000","mapping_epoch":15,"log_start":"153'511863","ondisk_log_start":"153'511863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1148,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 158s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6975860755,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":514555,"num_read_kb":2027300,"num_write":514434,"num_write_kb":2057730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'18696","reported_seq":24349,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:32.313635+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:55:32.313635+0000","last_peered":"2023-10-04T01:55:32.313635+0000","last_clean":"2023-10-04T01:55:32.313635+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:55:32.313635+0000","last_undegraded":"2023-10-04T01:55:32.313635+0000","last_fullsized":"2023-10-04T01:55:32.313635+0000","mapping_epoch":12,"log_start":"95'8600","ondisk_log_start":"95'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":333,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-04T01:54:18.409205+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:54:18.409205+0000","last_peered":"2023-10-04T01:54:18.409205+0000","last_clean":"2023-10-04T01:54:18.409205+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:54:18.409205+0000","last_undegraded":"2023-10-04T01:54:18.409205+0000","last_fullsized":"2023-10-04T01:54:18.409205+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28381593619,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2093023,"num_read_kb":8247304,"num_write":2090940,"num_write_kb":8363754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17127976960,"data_stored":36815073318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40207,"ondisk_log_size":40207,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59356,"ondisk_log_size":59356,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739358,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10361400,"kb_used_data":8363952,"kb_used_omap":2628,"kb_used_meta":1994811,"kb_avail":84010440,"statfs":{"total":96636764160,"available":86026690560,"internally_reserved":0,"allocated":8564686848,"data_stored":18406925547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":2042687367},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:55:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":4.4580000000000002,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.492,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Oct 4 01:54:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.749,"15min":0.58199999999999996},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.349,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.71999999999999997,"15min":0.56399999999999995},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":0.38800000000000001,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5483164,"kb_used_data":4117844,"kb_used_omap":1191,"kb_used_meta":1364120,"kb_avail":88888676,"statfs":{"total":96636764160,"available":91022004224,"internally_reserved":0,"allocated":4216672256,"data_stored":9066408571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1396859257},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2,3,8,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.313,"15min":0.33900000000000002},"min":{"1min":0.189,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.52800000000000002},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":4.2699999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.51700000000000002,"15min":0.58699999999999997},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.96,"5min":4.641,"15min":10.494999999999999},"last":0.254},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.49299999999999999,"15min":0.56000000000000005},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5688772,"kb_used_data":4251324,"kb_used_omap":1357,"kb_used_meta":1436082,"kb_avail":88683068,"statfs":{"total":96636764160,"available":90811461632,"internally_reserved":0,"allocated":4353355776,"data_stored":9343897042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1470548860},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,8,2,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:54:37 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41899999999999998,"15min":0.48899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.2130000000000001,"5min":4.0730000000000004,"15min":17.991},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.52800000000000002,"15min":0.60999999999999999},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":0.76200000000000001,"5min":5.6210000000000004,"15min":17.957000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:54:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.73099999999999998,"15min":0.65800000000000003},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.56499999999999995,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.375,"15min":0.46600000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":2.597,"5min":2.597,"15min":14.051},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4350869504,"data_stored":9342844928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4213616640,"data_stored":9065189395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8563490816,"data_stored":18407038995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:55:37.370 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:55:37.370 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:55:57.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:55:58.074 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:55:58.075 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:55:58.107 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2636,"stamp":"2023-10-04T01:55:57.789279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28418757177,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2313148,"num_read_kb":8459797,"num_write":2155507,"num_write_kb":8455567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99735,"ondisk_log_size":99735,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21224572,"kb_used_data":16807204,"kb_used_omap":5176,"kb_used_meta":4412167,"kb_avail":261890948,"statfs":{"total":289910292480,"available":268176330752,"internally_reserved":0,"allocated":17210576896,"data_stored":36893093176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4518059132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,6,5,6,21,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20144128,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5071,"num_read_kb":20220,"num_write":4927,"num_write_kb":19708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001911"},"pg_stats":[{"pgid":"2.7","version":"155'8775","reported_seq":12338,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.719020+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:55:53.719020+0000","last_peered":"2023-10-04T01:55:53.719020+0000","last_clean":"2023-10-04T01:55:53.719020+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:55:53.719020+0000","last_undegraded":"2023-10-04T01:55:53.719020+0000","last_fullsized":"2023-10-04T01:55:53.719020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'7641","reported_seq":11429,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.715983+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:55:53.715983+0000","last_peered":"2023-10-04T01:55:53.715983+0000","last_clean":"2023-10-04T01:55:53.715983+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:55:53.715983+0000","last_undegraded":"2023-10-04T01:55:53.715983+0000","last_fullsized":"2023-10-04T01:55:53.715983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7641,"log_dups_size":0,"ondisk_log_size":7641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'8921","reported_seq":12792,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.720043+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:55:53.720043+0000","last_peered":"2023-10-04T01:55:53.720043+0000","last_clean":"2023-10-04T01:55:53.720043+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:55:53.720043+0000","last_undegraded":"2023-10-04T01:55:53.720043+0000","last_fullsized":"2023-10-04T01:55:53.720043+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'7350","reported_seq":12067,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.724643+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:55:53.724643+0000","last_peered":"2023-10-04T01:55:53.724643+0000","last_clean":"2023-10-04T01:55:53.724643+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:55:53.724643+0000","last_undegraded":"2023-10-04T01:55:53.724643+0000","last_fullsized":"2023-10-04T01:55:53.724643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7350,"log_dups_size":0,"ondisk_log_size":7350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'539473","reported_seq":1063003,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:57.150307+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:55:57.150307+0000","last_peered":"2023-10-04T01:55:57.150307+0000","last_clean":"2023-10-04T01:55:57.150307+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:55:57.150307+0000","last_undegraded":"2023-10-04T01:55:57.150307+0000","last_fullsized":"2023-10-04T01:55:57.150307+0000","mapping_epoch":15,"log_start":"153'529373","ondisk_log_start":"153'529373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7136972800,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":528431,"num_read_kb":2082136,"num_write":527924,"num_write_kb":2111696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'17826","reported_seq":23267,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:54.379214+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:55:54.379214+0000","last_peered":"2023-10-04T01:55:54.379214+0000","last_clean":"2023-10-04T01:55:54.379214+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:55:54.379214+0000","last_undegraded":"2023-10-04T01:55:54.379214+0000","last_fullsized":"2023-10-04T01:55:54.379214+0000","mapping_epoch":12,"log_start":"93'7800","ondisk_log_start":"93'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'527652","reported_seq":1039447,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.972254+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:55:53.972254+0000","last_peered":"2023-10-04T01:55:53.972254+0000","last_clean":"2023-10-04T01:55:53.972254+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:55:53.972254+0000","last_undegraded":"2023-10-04T01:55:53.972254+0000","last_fullsized":"2023-10-04T01:55:53.972254+0000","mapping_epoch":15,"log_start":"153'517567","ondisk_log_start":"153'517567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6987653120,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":516620,"num_read_kb":2035580,"num_write":516272,"num_write_kb":2065088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9403,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:54.387038+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:55:54.387038+0000","last_peered":"2023-10-04T01:55:54.387038+0000","last_clean":"2023-10-04T01:55:54.387038+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:55:54.387038+0000","last_undegraded":"2023-10-04T01:55:54.387038+0000","last_fullsized":"2023-10-04T01:55:54.387038+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'551180","reported_seq":1086661,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:57.150901+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:55:57.150901+0000","last_peered":"2023-10-04T01:55:57.150901+0000","last_clean":"2023-10-04T01:55:57.150901+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:55:57.150901+0000","last_undegraded":"2023-10-04T01:55:57.150901+0000","last_fullsized":"2023-10-04T01:55:57.150901+0000","mapping_epoch":15,"log_start":"154'541166","ondisk_log_start":"154'541166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7309811712,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":540495,"num_read_kb":2130472,"num_write":539319,"num_write_kb":2157276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9856,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:54.381008+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:55:54.381008+0000","last_peered":"2023-10-04T01:55:54.381008+0000","last_clean":"2023-10-04T01:55:54.381008+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:55:54.381008+0000","last_undegraded":"2023-10-04T01:55:54.381008+0000","last_fullsized":"2023-10-04T01:55:54.381008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'523903","reported_seq":1035407,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:55:53.976601+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:55:53.976601+0000","last_peered":"2023-10-04T01:55:53.976601+0000","last_clean":"2023-10-04T01:55:53.976601+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:55:53.976601+0000","last_undegraded":"2023-10-04T01:55:53.976601+0000","last_fullsized":"2023-10-04T01:55:53.976601+0000","mapping_epoch":15,"log_start":"153'513863","ondisk_log_start":"153'513863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1286,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 178s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6983712787,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":516428,"num_read_kb":2034756,"num_write":516348,"num_write_kb":2065386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'18753","reported_seq":24418,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:53.761383+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:55:53.761383+0000","last_peered":"2023-10-04T01:55:53.761383+0000","last_clean":"2023-10-04T01:55:53.761383+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:55:53.761383+0000","last_undegraded":"2023-10-04T01:55:53.761383+0000","last_fullsized":"2023-10-04T01:55:53.761383+0000","mapping_epoch":12,"log_start":"95'8700","ondisk_log_start":"95'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":334,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:28.415062+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:28.415062+0000","last_peered":"2023-10-04T01:55:28.415062+0000","last_clean":"2023-10-04T01:55:28.415062+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:28.415062+0000","last_undegraded":"2023-10-04T01:55:28.415062+0000","last_fullsized":"2023-10-04T01:55:28.415062+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28418150419,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2101974,"num_read_kb":8282944,"num_write":2099863,"num_write_kb":8399446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17204953088,"data_stored":36892049446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40239,"ondisk_log_size":40239,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59397,"ondisk_log_size":59397,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739363,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9896940,"kb_used_data":8403108,"kb_used_omap":2628,"kb_used_meta":1491195,"kb_avail":84474900,"statfs":{"total":96636764160,"available":86502297600,"internally_reserved":0,"allocated":8604782592,"data_stored":18447021291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1526984583},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:55:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":4.4580000000000002,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.492,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Oct 4 01:55:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.68300000000000005,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.65000000000000002,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.374},{"interface":"front","average":{"1min":0.503,"5min":0.75700000000000001,"15min":0.57399999999999995},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":3.706,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359739360,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5534100,"kb_used_data":4132684,"kb_used_omap":1191,"kb_used_meta":1400216,"kb_avail":88837740,"statfs":{"total":96636764160,"available":90969845760,"internally_reserved":0,"allocated":4231868416,"data_stored":9081604731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1433821561},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,2,6,21,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.313,"15min":0.33900000000000002},"min":{"1min":0.189,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.52800000000000002},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":4.2699999999999996,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.49099999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:54:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.51700000000000002,"15min":0.58699999999999997},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":3.96,"5min":4.641,"15min":10.494999999999999},"last":0.217},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.49299999999999999,"15min":0.56000000000000005},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.20599999999999999},"max":{"1min":4.0949999999999998,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5793532,"kb_used_data":4271412,"kb_used_omap":1357,"kb_used_meta":1520754,"kb_avail":88578308,"statfs":{"total":96636764160,"available":90704187392,"internally_reserved":0,"allocated":4373925888,"data_stored":9364467154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1557252988},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.6859999999999999,"5min":4.0730000000000004,"15min":17.991},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.61199999999999999},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":1.7569999999999999,"5min":4.1139999999999999,"15min":17.957000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:54:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.73099999999999998,"15min":0.65800000000000003},"min":{"1min":0.29199999999999998,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.56499999999999995,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.375,"15min":0.46600000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20200000000000001},"max":{"1min":2.597,"5min":2.597,"15min":14.051},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4371435520,"data_stored":9363410944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4229627904,"data_stored":9081200659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8603889664,"data_stored":18447437843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:55:58.109 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:55:58.110 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:56:04.622 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:04.616+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:56:04.623 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:04.616+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:56:18.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:56:19.045 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:56:19.046 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:56:19.083 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2646,"stamp":"2023-10-04T01:56:17.795447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28451172921,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2321179,"num_read_kb":8491753,"num_write":2163400,"num_write_kb":8487139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99843,"ondisk_log_size":99843,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21394812,"kb_used_data":16870820,"kb_used_omap":5176,"kb_used_meta":4518791,"kb_avail":261720708,"statfs":{"total":289910292480,"available":268002004992,"internally_reserved":0,"allocated":17275719680,"data_stored":36958235960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4627242108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,6,7,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20889600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5186,"num_read_kb":20632,"num_write":5083,"num_write_kb":20332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004794"},"pg_stats":[{"pgid":"2.7","version":"155'8801","reported_seq":12373,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:09.743051+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:56:09.743051+0000","last_peered":"2023-10-04T01:56:09.743051+0000","last_clean":"2023-10-04T01:56:09.743051+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:56:09.743051+0000","last_undegraded":"2023-10-04T01:56:09.743051+0000","last_fullsized":"2023-10-04T01:56:09.743051+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8801,"log_dups_size":0,"ondisk_log_size":8801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'7654","reported_seq":11446,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:09.670761+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:56:09.670761+0000","last_peered":"2023-10-04T01:56:09.670761+0000","last_clean":"2023-10-04T01:56:09.670761+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:56:09.670761+0000","last_undegraded":"2023-10-04T01:56:09.670761+0000","last_fullsized":"2023-10-04T01:56:09.670761+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'8948","reported_seq":12827,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:09.742639+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:56:09.742639+0000","last_peered":"2023-10-04T01:56:09.742639+0000","last_clean":"2023-10-04T01:56:09.742639+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:56:09.742639+0000","last_undegraded":"2023-10-04T01:56:09.742639+0000","last_fullsized":"2023-10-04T01:56:09.742639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8948,"log_dups_size":0,"ondisk_log_size":8948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'7377","reported_seq":12102,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:09.743036+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:56:09.743036+0000","last_peered":"2023-10-04T01:56:09.743036+0000","last_clean":"2023-10-04T01:56:09.743036+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:56:09.743036+0000","last_undegraded":"2023-10-04T01:56:09.743036+0000","last_fullsized":"2023-10-04T01:56:09.743036+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7377,"log_dups_size":0,"ondisk_log_size":7377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'541483","reported_seq":1067029,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:17.241466+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:56:17.241466+0000","last_peered":"2023-10-04T01:56:17.241466+0000","last_clean":"2023-10-04T01:56:17.241466+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:56:17.241466+0000","last_undegraded":"2023-10-04T01:56:17.241466+0000","last_fullsized":"2023-10-04T01:56:17.241466+0000","mapping_epoch":15,"log_start":"154'531473","ondisk_log_start":"154'531473","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7145230336,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":530446,"num_read_kb":2090128,"num_write":529934,"num_write_kb":2119736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'17883","reported_seq":23336,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:16.593136+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:56:16.593136+0000","last_peered":"2023-10-04T01:56:16.593136+0000","last_clean":"2023-10-04T01:56:16.593136+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:56:16.593136+0000","last_undegraded":"2023-10-04T01:56:16.593136+0000","last_fullsized":"2023-10-04T01:56:16.593136+0000","mapping_epoch":12,"log_start":"93'7800","ondisk_log_start":"93'7800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'529555","reported_seq":1043302,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:14.096901+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:56:14.096901+0000","last_peered":"2023-10-04T01:56:14.096901+0000","last_clean":"2023-10-04T01:56:14.096901+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:56:14.096901+0000","last_undegraded":"2023-10-04T01:56:14.096901+0000","last_fullsized":"2023-10-04T01:56:14.096901+0000","mapping_epoch":15,"log_start":"154'519467","ondisk_log_start":"154'519467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":6995439616,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":518568,"num_read_kb":2043320,"num_write":518175,"num_write_kb":2072700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9419,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:14.917109+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:56:14.917109+0000","last_peered":"2023-10-04T01:56:14.917109+0000","last_clean":"2023-10-04T01:56:14.917109+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:56:14.917109+0000","last_undegraded":"2023-10-04T01:56:14.917109+0000","last_fullsized":"2023-10-04T01:56:14.917109+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'553207","reported_seq":1090820,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:17.232951+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:56:17.232951+0000","last_peered":"2023-10-04T01:56:17.232951+0000","last_clean":"2023-10-04T01:56:17.232951+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:56:17.232951+0000","last_undegraded":"2023-10-04T01:56:17.232951+0000","last_fullsized":"2023-10-04T01:56:17.232951+0000","mapping_epoch":15,"log_start":"154'543166","ondisk_log_start":"154'543166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7318151168,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":542625,"num_read_kb":2138956,"num_write":541346,"num_write_kb":2165384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9868,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:14.859938+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:56:14.859938+0000","last_peered":"2023-10-04T01:56:14.859938+0000","last_clean":"2023-10-04T01:56:14.859938+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:56:14.859938+0000","last_undegraded":"2023-10-04T01:56:14.859938+0000","last_fullsized":"2023-10-04T01:56:14.859938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'525981","reported_seq":1039432,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:56:14.124045+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:56:14.124045+0000","last_peered":"2023-10-04T01:56:14.124045+0000","last_clean":"2023-10-04T01:56:14.124045+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:56:14.124045+0000","last_undegraded":"2023-10-04T01:56:14.124045+0000","last_fullsized":"2023-10-04T01:56:14.124045+0000","mapping_epoch":15,"log_start":"154'515963","ondisk_log_start":"154'515963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1438,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6991745043,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":518366,"num_read_kb":2042496,"num_write":518301,"num_write_kb":2073198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'18793","reported_seq":24470,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:09.742917+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:56:09.742917+0000","last_peered":"2023-10-04T01:56:09.742917+0000","last_clean":"2023-10-04T01:56:09.742917+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:56:09.742917+0000","last_undegraded":"2023-10-04T01:56:09.742917+0000","last_fullsized":"2023-10-04T01:56:09.742917+0000","mapping_epoch":12,"log_start":"95'8700","ondisk_log_start":"95'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":334,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:28.415062+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:28.415062+0000","last_peered":"2023-10-04T01:55:28.415062+0000","last_clean":"2023-10-04T01:55:28.415062+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:28.415062+0000","last_undegraded":"2023-10-04T01:55:28.415062+0000","last_fullsized":"2023-10-04T01:55:28.415062+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28450566163,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2110005,"num_read_kb":8314900,"num_write":2107756,"num_write_kb":8431018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17269608448,"data_stored":36956704806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40157,"ondisk_log_size":40157,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59587,"ondisk_log_size":59587,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739367,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9962876,"kb_used_data":8434612,"kb_used_omap":2628,"kb_used_meta":1525627,"kb_avail":84408964,"statfs":{"total":96636764160,"available":86434779136,"internally_reserved":0,"allocated":8637042688,"data_stored":18479281387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1562242951},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.17299999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.53700000000000003,"15min":0.54500000000000004},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.61599999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:55:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.68300000000000005,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.65000000000000002,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.75700000000000001,"15min":0.57399999999999995},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":3.706,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.254}]}]},{"osd":1,"up_from":8,"seq":34359739364,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5586360,"kb_used_data":4148848,"kb_used_omap":1191,"kb_used_meta":1436312,"kb_avail":88785480,"statfs":{"total":96636764160,"available":90916331520,"internally_reserved":0,"allocated":4248420352,"data_stored":9098156667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1470783865},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.311,"15min":0.34000000000000002},"min":{"1min":0.20000000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.46000000000000002,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.249},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.59299999999999997,"15min":0.53500000000000003},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":2.1909999999999998,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.28799999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.60299999999999998},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.7730000000000001,"5min":4.641,"15min":10.494999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.57499999999999996},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.7269999999999999,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359739365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5845576,"kb_used_data":4287360,"kb_used_omap":1357,"kb_used_meta":1556850,"kb_avail":88526264,"statfs":{"total":96636764160,"available":90650894336,"internally_reserved":0,"allocated":4390256640,"data_stored":9380797906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1594215292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.6859999999999999,"5min":4.0730000000000004,"15min":17.991},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.61199999999999999},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":1.7569999999999999,"5min":4.1139999999999999,"15min":17.957000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:55:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.71499999999999997,"15min":0.65200000000000002},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":1.72,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.375,"15min":0.46400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.57799999999999996,"5min":2.597,"15min":14.051},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4387971072,"data_stored":9379946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4245413888,"data_stored":9096986643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8636223488,"data_stored":18479771667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:56:19.085 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:56:19.085 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:56:39.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:56:39.576 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:56:39.576 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:56:39.605 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2657,"stamp":"2023-10-04T01:56:38.420935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28483297849,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2329261,"num_read_kb":8523913,"num_write":2171234,"num_write_kb":8518475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99845,"ondisk_log_size":99845,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21562068,"kb_used_data":16932988,"kb_used_omap":5176,"kb_used_meta":4623879,"kb_avail":261553452,"statfs":{"total":289910292480,"available":267830734848,"internally_reserved":0,"allocated":17339379712,"data_stored":37021895992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4734852220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,12,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15843328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4075,"num_read_kb":16192,"num_write":3860,"num_write_kb":15440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.623914"},"pg_stats":[{"pgid":"2.7","version":"155'8815","reported_seq":12395,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:31.462239+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:56:31.462239+0000","last_peered":"2023-10-04T01:56:31.462239+0000","last_clean":"2023-10-04T01:56:31.462239+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:56:31.462239+0000","last_undegraded":"2023-10-04T01:56:31.462239+0000","last_fullsized":"2023-10-04T01:56:31.462239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'7661","reported_seq":11457,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:30.623160+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:56:30.623160+0000","last_peered":"2023-10-04T01:56:30.623160+0000","last_clean":"2023-10-04T01:56:30.623160+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:56:30.623160+0000","last_undegraded":"2023-10-04T01:56:30.623160+0000","last_fullsized":"2023-10-04T01:56:30.623160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7661,"log_dups_size":0,"ondisk_log_size":7661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'8962","reported_seq":12849,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:31.457705+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:56:31.457705+0000","last_peered":"2023-10-04T01:56:31.457705+0000","last_clean":"2023-10-04T01:56:31.457705+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:56:31.457705+0000","last_undegraded":"2023-10-04T01:56:31.457705+0000","last_fullsized":"2023-10-04T01:56:31.457705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'7391","reported_seq":12124,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:31.457328+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:56:31.457328+0000","last_peered":"2023-10-04T01:56:31.457328+0000","last_clean":"2023-10-04T01:56:31.457328+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:56:31.457328+0000","last_undegraded":"2023-10-04T01:56:31.457328+0000","last_fullsized":"2023-10-04T01:56:31.457328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7391,"log_dups_size":0,"ondisk_log_size":7391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'543400","reported_seq":1071031,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:37.351520+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:56:37.351520+0000","last_peered":"2023-10-04T01:56:37.351520+0000","last_clean":"2023-10-04T01:56:37.351520+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:56:37.351520+0000","last_undegraded":"2023-10-04T01:56:37.351520+0000","last_fullsized":"2023-10-04T01:56:37.351520+0000","mapping_epoch":15,"log_start":"154'533373","ondisk_log_start":"154'533373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7153119232,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":532530,"num_read_kb":2098396,"num_write":531851,"num_write_kb":2127404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"155'17913","reported_seq":23378,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:36.932015+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:56:36.932015+0000","last_peered":"2023-10-04T01:56:36.932015+0000","last_clean":"2023-10-04T01:56:36.932015+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:56:36.932015+0000","last_undegraded":"2023-10-04T01:56:36.932015+0000","last_fullsized":"2023-10-04T01:56:36.932015+0000","mapping_epoch":12,"log_start":"95'7900","ondisk_log_start":"95'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'531524","reported_seq":1047270,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:34.219201+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:56:34.219201+0000","last_peered":"2023-10-04T01:56:34.219201+0000","last_clean":"2023-10-04T01:56:34.219201+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:56:34.219201+0000","last_undegraded":"2023-10-04T01:56:34.219201+0000","last_fullsized":"2023-10-04T01:56:34.219201+0000","mapping_epoch":15,"log_start":"154'521467","ondisk_log_start":"154'521467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7003492352,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":520566,"num_read_kb":2051292,"num_write":520144,"num_write_kb":2080576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9435,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:35.339781+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:56:35.339781+0000","last_peered":"2023-10-04T01:56:35.339781+0000","last_clean":"2023-10-04T01:56:35.339781+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:56:35.339781+0000","last_undegraded":"2023-10-04T01:56:35.339781+0000","last_fullsized":"2023-10-04T01:56:35.339781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'555256","reported_seq":1094898,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:37.353712+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:56:37.353712+0000","last_peered":"2023-10-04T01:56:37.353712+0000","last_clean":"2023-10-04T01:56:37.353712+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:56:37.353712+0000","last_undegraded":"2023-10-04T01:56:37.353712+0000","last_fullsized":"2023-10-04T01:56:37.353712+0000","mapping_epoch":15,"log_start":"154'545166","ondisk_log_start":"154'545166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7326539776,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":544651,"num_read_kb":2147028,"num_write":543395,"num_write_kb":2173580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9880,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:35.338877+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:56:35.338877+0000","last_peered":"2023-10-04T01:56:35.338877+0000","last_clean":"2023-10-04T01:56:35.338877+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:56:35.338877+0000","last_undegraded":"2023-10-04T01:56:35.338877+0000","last_fullsized":"2023-10-04T01:56:35.338877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'527948","reported_seq":1043386,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:56:34.224665+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:56:34.224665+0000","last_peered":"2023-10-04T01:56:34.224665+0000","last_clean":"2023-10-04T01:56:34.224665+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:56:34.224665+0000","last_undegraded":"2023-10-04T01:56:34.224665+0000","last_fullsized":"2023-10-04T01:56:34.224665+0000","mapping_epoch":15,"log_start":"154'517863","ondisk_log_start":"154'517863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1523,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 218s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":6999539731,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":520340,"num_read_kb":2050344,"num_write":520200,"num_write_kb":2080794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'18814","reported_seq":24503,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:56:31.460459+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:56:31.460459+0000","last_peered":"2023-10-04T01:56:31.460459+0000","last_clean":"2023-10-04T01:56:31.460459+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:56:31.460459+0000","last_undegraded":"2023-10-04T01:56:31.460459+0000","last_fullsized":"2023-10-04T01:56:31.460459+0000","mapping_epoch":12,"log_start":"95'8800","ondisk_log_start":"95'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":334,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:28.415062+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:28.415062+0000","last_peered":"2023-10-04T01:55:28.415062+0000","last_clean":"2023-10-04T01:55:28.415062+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:28.415062+0000","last_undegraded":"2023-10-04T01:55:28.415062+0000","last_fullsized":"2023-10-04T01:55:28.415062+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28482691091,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2118087,"num_read_kb":8347060,"num_write":2115590,"num_write_kb":8462354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17333735424,"data_stored":37020831782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40259,"ondisk_log_size":40259,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59487,"ondisk_log_size":59487,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10024572,"kb_used_data":8465972,"kb_used_omap":2628,"kb_used_meta":1555963,"kb_avail":84347268,"statfs":{"total":96636764160,"available":86371602432,"internally_reserved":0,"allocated":8669155328,"data_stored":18511394027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1593307015},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.17299999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.315},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.53700000000000003,"15min":0.54500000000000004},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.61599999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.34899999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:55:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.68300000000000005,"15min":0.58299999999999996},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.65000000000000002,"5min":33.975000000000001,"15min":33.975000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.75700000000000001,"15min":0.57399999999999995},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":3.706,"5min":35.423999999999999,"15min":35.423999999999999},"last":0.25800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5635212,"kb_used_data":4163716,"kb_used_omap":1191,"kb_used_meta":1470296,"kb_avail":88736628,"statfs":{"total":96636764160,"available":90866307072,"internally_reserved":0,"allocated":4263645184,"data_stored":9113381499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1505583481},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.311,"15min":0.34000000000000002},"min":{"1min":0.20000000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.46000000000000002,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.59299999999999997,"15min":0.53500000000000003},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":2.1909999999999998,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.60299999999999998},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.7730000000000001,"5min":4.641,"15min":10.494999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.57499999999999996},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.7269999999999999,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.33600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5902284,"kb_used_data":4303300,"kb_used_omap":1357,"kb_used_meta":1597618,"kb_avail":88469556,"statfs":{"total":96636764160,"available":90592825344,"internally_reserved":0,"allocated":4406579200,"data_stored":9397120466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1635961724},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:55:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.497},"min":{"1min":0.20599999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.6859999999999999,"5min":4.0730000000000004,"15min":17.991},"last":0.247},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.61199999999999999},"min":{"1min":0.251,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":1.7569999999999999,"5min":4.1139999999999999,"15min":17.957000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:55:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.71499999999999997,"15min":0.65200000000000002},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":1.72,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.375,"15min":0.46400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.57799999999999996,"5min":2.597,"15min":14.051},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4404207616,"data_stored":9396183040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4261253120,"data_stored":9112825875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8668274688,"data_stored":18511822867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:56:39.607 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:56:39.607 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:56:44.236 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:44.227+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:56:44.250 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:44.243+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:56:44.250 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:44.243+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:56:44.250 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:56:44.243+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:56:59.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:57:00.144 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:57:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:57:00.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2668,"stamp":"2023-10-04T01:56:59.808751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28519121465,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2338430,"num_read_kb":8560365,"num_write":2179957,"num_write_kb":8553367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99982,"ondisk_log_size":99982,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21852860,"kb_used_data":16999204,"kb_used_omap":5176,"kb_used_meta":4848455,"kb_avail":261262660,"statfs":{"total":289910292480,"available":267532963840,"internally_reserved":0,"allocated":17407184896,"data_stored":37089688876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":4964818044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,11,4,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19759104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5122,"num_read_kb":20340,"num_write":4806,"num_write_kb":19224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004331"},"pg_stats":[{"pgid":"2.7","version":"156'8854","reported_seq":12445,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.818967+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:56:55.818967+0000","last_peered":"2023-10-04T01:56:55.818967+0000","last_clean":"2023-10-04T01:56:55.818967+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:56:55.818967+0000","last_undegraded":"2023-10-04T01:56:55.818967+0000","last_fullsized":"2023-10-04T01:56:55.818967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'7682","reported_seq":11484,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.812024+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:56:55.812024+0000","last_peered":"2023-10-04T01:56:55.812024+0000","last_clean":"2023-10-04T01:56:55.812024+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:56:55.812024+0000","last_undegraded":"2023-10-04T01:56:55.812024+0000","last_fullsized":"2023-10-04T01:56:55.812024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'9002","reported_seq":12900,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.818785+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:56:55.818785+0000","last_peered":"2023-10-04T01:56:55.818785+0000","last_clean":"2023-10-04T01:56:55.818785+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:56:55.818785+0000","last_undegraded":"2023-10-04T01:56:55.818785+0000","last_fullsized":"2023-10-04T01:56:55.818785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'7431","reported_seq":12175,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.818761+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:56:55.818761+0000","last_peered":"2023-10-04T01:56:55.818761+0000","last_clean":"2023-10-04T01:56:55.818761+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:56:55.818761+0000","last_undegraded":"2023-10-04T01:56:55.818761+0000","last_fullsized":"2023-10-04T01:56:55.818761+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7431,"log_dups_size":0,"ondisk_log_size":7431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'545389","reported_seq":1075077,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:57.449241+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:56:57.449241+0000","last_peered":"2023-10-04T01:56:57.449241+0000","last_clean":"2023-10-04T01:56:57.449241+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:56:57.449241+0000","last_undegraded":"2023-10-04T01:56:57.449241+0000","last_fullsized":"2023-10-04T01:56:57.449241+0000","mapping_epoch":15,"log_start":"154'535373","ondisk_log_start":"154'535373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7161290752,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":534593,"num_read_kb":2106592,"num_write":533840,"num_write_kb":2135360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'17943","reported_seq":23421,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:56.673796+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:56:56.673796+0000","last_peered":"2023-10-04T01:56:56.673796+0000","last_clean":"2023-10-04T01:56:56.673796+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:56:56.673796+0000","last_undegraded":"2023-10-04T01:56:56.673796+0000","last_fullsized":"2023-10-04T01:56:56.673796+0000","mapping_epoch":12,"log_start":"95'7900","ondisk_log_start":"95'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'533854","reported_seq":1052112,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:59.385667+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:56:59.385667+0000","last_peered":"2023-10-04T01:56:59.385667+0000","last_clean":"2023-10-04T01:56:59.385667+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:56:59.385667+0000","last_undegraded":"2023-10-04T01:56:59.385667+0000","last_fullsized":"2023-10-04T01:56:59.385667+0000","mapping_epoch":15,"log_start":"154'523767","ondisk_log_start":"154'523767","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7013048320,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":523080,"num_read_kb":2061300,"num_write":522474,"num_write_kb":2089896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"140'2099","reported_seq":9453,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.819722+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:56:55.819722+0000","last_peered":"2023-10-04T01:56:55.819722+0000","last_clean":"2023-10-04T01:56:55.819722+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:56:55.819722+0000","last_undegraded":"2023-10-04T01:56:55.819722+0000","last_fullsized":"2023-10-04T01:56:55.819722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'557270","reported_seq":1099003,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:57.447468+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:56:57.447468+0000","last_peered":"2023-10-04T01:56:57.447468+0000","last_clean":"2023-10-04T01:56:57.447468+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:56:57.447468+0000","last_undegraded":"2023-10-04T01:56:57.447468+0000","last_fullsized":"2023-10-04T01:56:57.447468+0000","mapping_epoch":15,"log_start":"154'547266","ondisk_log_start":"154'547266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7334830080,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":546745,"num_read_kb":2155356,"num_write":545409,"num_write_kb":2181636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"141'4532","reported_seq":9893,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.818880+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:56:55.818880+0000","last_peered":"2023-10-04T01:56:55.818880+0000","last_clean":"2023-10-04T01:56:55.818880+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:56:55.818880+0000","last_undegraded":"2023-10-04T01:56:55.818880+0000","last_fullsized":"2023-10-04T01:56:55.818880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'530523","reported_seq":1048476,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:56:59.394541+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:56:59.394541+0000","last_peered":"2023-10-04T01:56:59.394541+0000","last_clean":"2023-10-04T01:56:59.394541+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:56:59.394541+0000","last_undegraded":"2023-10-04T01:56:59.394541+0000","last_fullsized":"2023-10-04T01:56:59.394541+0000","mapping_epoch":15,"log_start":"154'520463","ondisk_log_start":"154'520463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1753,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7009345555,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":522838,"num_read_kb":2060264,"num_write":522590,"num_write_kb":2090354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'18873","reported_seq":24578,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:56:55.824292+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:56:55.824292+0000","last_peered":"2023-10-04T01:56:55.824292+0000","last_clean":"2023-10-04T01:56:55.824292+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:56:55.824292+0000","last_undegraded":"2023-10-04T01:56:55.824292+0000","last_fullsized":"2023-10-04T01:56:55.824292+0000","mapping_epoch":12,"log_start":"95'8800","ondisk_log_start":"95'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":335,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:43.925275+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:43.925275+0000","last_peered":"2023-10-04T01:55:43.925275+0000","last_clean":"2023-10-04T01:55:43.925275+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:43.925275+0000","last_undegraded":"2023-10-04T01:55:43.925275+0000","last_fullsized":"2023-10-04T01:55:43.925275+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28518514707,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2127256,"num_read_kb":8383512,"num_write":2124313,"num_write_kb":8497246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17401241600,"data_stored":37088337958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40167,"ondisk_log_size":40167,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59716,"ondisk_log_size":59716,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739375,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10159452,"kb_used_data":8496724,"kb_used_omap":2628,"kb_used_meta":1660091,"kb_avail":84212388,"statfs":{"total":96636764160,"available":86233485312,"internally_reserved":0,"allocated":8700645376,"data_stored":18542879975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1699934087},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.17299999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.57999999999999996,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.53700000000000003,"15min":0.54500000000000004},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.61599999999999999,"5min":6.7930000000000001,"15min":6.7930000000000001},"last":0.432}]},{"osd":1,"last update":"Wed Oct 4 01:56:51 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.374,"15min":0.58199999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.74399999999999999,"5min":1.349,"15min":33.975000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.436,"15min":0.56599999999999995},"min":{"1min":0.23400000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.88,"5min":5.2240000000000002,"15min":35.423999999999999},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359739373,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5720960,"kb_used_data":4183160,"kb_used_omap":1191,"kb_used_meta":1536600,"kb_avail":88650880,"statfs":{"total":96636764160,"available":90778501120,"internally_reserved":0,"allocated":4283555840,"data_stored":9133288055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1573478777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.311,"15min":0.34000000000000002},"min":{"1min":0.20000000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.46000000000000002,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.59299999999999997,"15min":0.53500000000000003},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.20300000000000001},"max":{"1min":2.1909999999999998,"5min":6.7169999999999996,"15min":6.7169999999999996},"last":0.89600000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.60299999999999998},"min":{"1min":0.217,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.7730000000000001,"5min":4.641,"15min":10.494999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.57499999999999996},"min":{"1min":0.24299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.7269999999999999,"5min":4.0949999999999998,"15min":10.531000000000001},"last":0.502}]}]},{"osd":0,"up_from":8,"seq":34359739373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5972448,"kb_used_data":4319320,"kb_used_omap":1357,"kb_used_meta":1651762,"kb_avail":88399392,"statfs":{"total":96636764160,"available":90520977408,"internally_reserved":0,"allocated":4422983680,"data_stored":9413520846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1691405180},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:56:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":17.991},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.51300000000000001,"15min":0.621},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:55:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.71499999999999997,"15min":0.65200000000000002},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":1.72,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.316},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.375,"15min":0.46400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.57799999999999996,"5min":2.597,"15min":14.051},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4420603904,"data_stored":9412579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4280578048,"data_stored":9132150803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8700059648,"data_stored":18543607827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:57:00.174 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:57:00.174 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:57:01.725 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:57:01.710+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:57:01.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:57:01.722+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:57:01.731 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:57:01.722+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:57:20.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:57:20.679 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:57:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:57:20.716 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2678,"stamp":"2023-10-04T01:57:19.812176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28551389753,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2346739,"num_read_kb":8593413,"num_write":2187823,"num_write_kb":8584831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100042,"ondisk_log_size":100042,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22124192,"kb_used_data":17063368,"kb_used_omap":5176,"kb_used_meta":5055623,"kb_avail":260991328,"statfs":{"total":289910292480,"available":267255119872,"internally_reserved":0,"allocated":17472888832,"data_stored":37155392812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":5176958076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,3,4,7,8,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20164608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5207,"num_read_kb":20720,"num_write":4916,"num_write_kb":19664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002109"},"pg_stats":[{"pgid":"2.7","version":"156'8880","reported_seq":12479,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:18.038161+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:57:18.038161+0000","last_peered":"2023-10-04T01:57:18.038161+0000","last_clean":"2023-10-04T01:57:18.038161+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:57:18.038161+0000","last_undegraded":"2023-10-04T01:57:18.038161+0000","last_fullsized":"2023-10-04T01:57:18.038161+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'7696","reported_seq":11502,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:16.298855+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:57:16.298855+0000","last_peered":"2023-10-04T01:57:16.298855+0000","last_clean":"2023-10-04T01:57:16.298855+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:57:16.298855+0000","last_undegraded":"2023-10-04T01:57:16.298855+0000","last_fullsized":"2023-10-04T01:57:16.298855+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'9029","reported_seq":12936,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:18.044117+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:57:18.044117+0000","last_peered":"2023-10-04T01:57:18.044117+0000","last_clean":"2023-10-04T01:57:18.044117+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:57:18.044117+0000","last_undegraded":"2023-10-04T01:57:18.044117+0000","last_fullsized":"2023-10-04T01:57:18.044117+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9029,"log_dups_size":0,"ondisk_log_size":9029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'7458","reported_seq":12211,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:18.026799+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:57:18.026799+0000","last_peered":"2023-10-04T01:57:18.026799+0000","last_clean":"2023-10-04T01:57:18.026799+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:57:18.026799+0000","last_undegraded":"2023-10-04T01:57:18.026799+0000","last_fullsized":"2023-10-04T01:57:18.026799+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'547437","reported_seq":1079250,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:17.516919+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:57:17.516919+0000","last_peered":"2023-10-04T01:57:17.516919+0000","last_clean":"2023-10-04T01:57:17.516919+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:57:17.516919+0000","last_undegraded":"2023-10-04T01:57:17.516919+0000","last_fullsized":"2023-10-04T01:57:17.516919+0000","mapping_epoch":15,"log_start":"155'537373","ondisk_log_start":"155'537373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7169716224,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":536711,"num_read_kb":2115012,"num_write":535888,"num_write_kb":2143552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'17976","reported_seq":23467,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:16.690218+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:57:16.690218+0000","last_peered":"2023-10-04T01:57:16.690218+0000","last_clean":"2023-10-04T01:57:16.690218+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:57:16.690218+0000","last_undegraded":"2023-10-04T01:57:16.690218+0000","last_fullsized":"2023-10-04T01:57:16.690218+0000","mapping_epoch":12,"log_start":"95'7900","ondisk_log_start":"95'7900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'535771","reported_seq":1056109,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:19.464315+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:57:19.464315+0000","last_peered":"2023-10-04T01:57:19.464315+0000","last_clean":"2023-10-04T01:57:19.464315+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:57:19.464315+0000","last_undegraded":"2023-10-04T01:57:19.464315+0000","last_fullsized":"2023-10-04T01:57:19.464315+0000","mapping_epoch":15,"log_start":"155'525767","ondisk_log_start":"155'525767","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7020949504,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":525156,"num_read_kb":2069576,"num_write":524391,"num_write_kb":2097564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9469,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:16.310168+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:57:16.310168+0000","last_peered":"2023-10-04T01:57:16.310168+0000","last_clean":"2023-10-04T01:57:16.310168+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:57:16.310168+0000","last_undegraded":"2023-10-04T01:57:16.310168+0000","last_fullsized":"2023-10-04T01:57:16.310168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'559223","reported_seq":1103089,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:17.511668+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:57:17.511668+0000","last_peered":"2023-10-04T01:57:17.511668+0000","last_clean":"2023-10-04T01:57:17.511668+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:57:17.511668+0000","last_undegraded":"2023-10-04T01:57:17.511668+0000","last_fullsized":"2023-10-04T01:57:17.511668+0000","mapping_epoch":15,"log_start":"155'549166","ondisk_log_start":"155'549166","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7342817280,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":548873,"num_read_kb":2163816,"num_write":547362,"num_write_kb":2189448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9905,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:16.301063+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:57:16.301063+0000","last_peered":"2023-10-04T01:57:16.301063+0000","last_clean":"2023-10-04T01:57:16.301063+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:57:16.301063+0000","last_undegraded":"2023-10-04T01:57:16.301063+0000","last_fullsized":"2023-10-04T01:57:16.301063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'532591","reported_seq":1052542,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:57:19.447084+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:57:19.447084+0000","last_peered":"2023-10-04T01:57:19.447084+0000","last_clean":"2023-10-04T01:57:19.447084+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:57:19.447084+0000","last_undegraded":"2023-10-04T01:57:19.447084+0000","last_fullsized":"2023-10-04T01:57:19.447084+0000","mapping_epoch":15,"log_start":"155'522563","ondisk_log_start":"155'522563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":1903,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 264s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7017299987,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":524825,"num_read_kb":2068156,"num_write":524538,"num_write_kb":2098146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'18913","reported_seq":24631,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:18.063406+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:57:18.063406+0000","last_peered":"2023-10-04T01:57:18.063406+0000","last_clean":"2023-10-04T01:57:18.063406+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:57:18.063406+0000","last_undegraded":"2023-10-04T01:57:18.063406+0000","last_fullsized":"2023-10-04T01:57:18.063406+0000","mapping_epoch":12,"log_start":"95'8900","ondisk_log_start":"95'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":335,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:43.925275+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:43.925275+0000","last_peered":"2023-10-04T01:55:43.925275+0000","last_clean":"2023-10-04T01:55:43.925275+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:43.925275+0000","last_undegraded":"2023-10-04T01:55:43.925275+0000","last_fullsized":"2023-10-04T01:55:43.925275+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28550782995,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2135565,"num_read_kb":8416560,"num_write":2132179,"num_write_kb":8528710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17465757696,"data_stored":37152854054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40153,"ondisk_log_size":40153,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59790,"ondisk_log_size":59790,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739379,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10310708,"kb_used_data":8529580,"kb_used_omap":2628,"kb_used_meta":1778491,"kb_avail":84061132,"statfs":{"total":96636764160,"available":86078599168,"internally_reserved":0,"allocated":8734289920,"data_stored":18576524519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1821175687},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:03 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.42799999999999999,"15min":0.46800000000000003},"min":{"1min":0.17100000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.71199999999999997,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.47699999999999998,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79000000000000004,"5min":4.492,"15min":6.7930000000000001},"last":0.374}]},{"osd":1,"last update":"Wed Oct 4 01:56:51 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.374,"15min":0.58199999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.74399999999999999,"5min":1.349,"15min":33.975000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.436,"15min":0.56599999999999995},"min":{"1min":0.23400000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.88,"5min":5.2240000000000002,"15min":35.423999999999999},"last":0.39300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739377,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5770960,"kb_used_data":4198536,"kb_used_omap":1191,"kb_used_meta":1571224,"kb_avail":88600880,"statfs":{"total":96636764160,"available":90727301120,"internally_reserved":0,"allocated":4299300864,"data_stored":9149033079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1608933753},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,5,7,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.314,"15min":0.33200000000000002},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.8460000000000001,"5min":6.431,"15min":6.7169999999999996},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.4219999999999997,"5min":4.641,"15min":10.494999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.57899999999999996},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.3730000000000002,"5min":4.3730000000000002,"15min":10.531000000000001},"last":0.33200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6042524,"kb_used_data":4335252,"kb_used_omap":1357,"kb_used_meta":1705906,"kb_avail":88329316,"statfs":{"total":96636764160,"available":90449219584,"internally_reserved":0,"allocated":4439298048,"data_stored":9429835214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1746848636},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:56:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":17.991},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.51300000000000001,"15min":0.621},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:56:59 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.80300000000000005,"15min":0.66100000000000003},"min":{"1min":0.26100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":5.4450000000000003,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.41199999999999998,"15min":0.46600000000000003},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":14.051},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4437004288,"data_stored":9428979712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4296413184,"data_stored":9147985939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8732340224,"data_stored":18575888403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:57:20.718 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:57:20.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:57:21.231 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:57:21.280 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:57:41.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:57:41.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:57:41.790 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:57:41.815 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2688,"stamp":"2023-10-04T01:57:39.815631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28583834169,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2354952,"num_read_kb":8626073,"num_write":2195734,"num_write_kb":8616475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100145,"ondisk_log_size":100145,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22434196,"kb_used_data":17126268,"kb_used_omap":5176,"kb_used_meta":5302727,"kb_avail":260681324,"statfs":{"total":289910292480,"available":266937675776,"internally_reserved":0,"allocated":17537298432,"data_stored":37219802412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":5429992572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,8,10,9,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20230144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5226,"num_read_kb":20796,"num_write":4945,"num_write_kb":19780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002078"},"pg_stats":[{"pgid":"2.7","version":"156'8908","reported_seq":12515,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:39.107647+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:57:39.107647+0000","last_peered":"2023-10-04T01:57:39.107647+0000","last_clean":"2023-10-04T01:57:39.107647+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:57:39.107647+0000","last_undegraded":"2023-10-04T01:57:39.107647+0000","last_fullsized":"2023-10-04T01:57:39.107647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8908,"log_dups_size":0,"ondisk_log_size":8908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'7710","reported_seq":11520,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:36.778843+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:57:36.778843+0000","last_peered":"2023-10-04T01:57:36.778843+0000","last_clean":"2023-10-04T01:57:36.778843+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:57:36.778843+0000","last_undegraded":"2023-10-04T01:57:36.778843+0000","last_fullsized":"2023-10-04T01:57:36.778843+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'9057","reported_seq":12972,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:39.108036+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:57:39.108036+0000","last_peered":"2023-10-04T01:57:39.108036+0000","last_clean":"2023-10-04T01:57:39.108036+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:57:39.108036+0000","last_undegraded":"2023-10-04T01:57:39.108036+0000","last_fullsized":"2023-10-04T01:57:39.108036+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'7486","reported_seq":12247,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:39.102768+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:57:39.102768+0000","last_peered":"2023-10-04T01:57:39.102768+0000","last_clean":"2023-10-04T01:57:39.102768+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:57:39.102768+0000","last_undegraded":"2023-10-04T01:57:39.102768+0000","last_fullsized":"2023-10-04T01:57:39.102768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7486,"log_dups_size":0,"ondisk_log_size":7486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'549452","reported_seq":1083363,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:37.626871+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:57:37.626871+0000","last_peered":"2023-10-04T01:57:37.626871+0000","last_clean":"2023-10-04T01:57:37.626871+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:57:37.626871+0000","last_undegraded":"2023-10-04T01:57:37.626871+0000","last_fullsized":"2023-10-04T01:57:37.626871+0000","mapping_epoch":15,"log_start":"155'539373","ondisk_log_start":"155'539373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7178006528,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":538808,"num_read_kb":2123352,"num_write":537903,"num_write_kb":2151612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'18030","reported_seq":23533,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:36.778790+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:57:36.778790+0000","last_peered":"2023-10-04T01:57:36.778790+0000","last_clean":"2023-10-04T01:57:36.778790+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:57:36.778790+0000","last_undegraded":"2023-10-04T01:57:36.778790+0000","last_fullsized":"2023-10-04T01:57:36.778790+0000","mapping_epoch":12,"log_start":"95'8000","ondisk_log_start":"95'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'537669","reported_seq":1059909,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:39.528073+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:57:39.528073+0000","last_peered":"2023-10-04T01:57:39.528073+0000","last_clean":"2023-10-04T01:57:39.528073+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:57:39.528073+0000","last_undegraded":"2023-10-04T01:57:39.528073+0000","last_fullsized":"2023-10-04T01:57:39.528073+0000","mapping_epoch":15,"log_start":"155'527667","ondisk_log_start":"155'527667","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7028682752,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":527060,"num_read_kb":2077156,"num_write":526289,"num_write_kb":2105156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9485,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:36.779777+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:57:36.779777+0000","last_peered":"2023-10-04T01:57:36.779777+0000","last_clean":"2023-10-04T01:57:36.779777+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:57:36.779777+0000","last_undegraded":"2023-10-04T01:57:36.779777+0000","last_fullsized":"2023-10-04T01:57:36.779777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'561277","reported_seq":1107314,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:37.639895+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:57:37.639895+0000","last_peered":"2023-10-04T01:57:37.639895+0000","last_clean":"2023-10-04T01:57:37.639895+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:57:37.639895+0000","last_undegraded":"2023-10-04T01:57:37.639895+0000","last_fullsized":"2023-10-04T01:57:37.639895+0000","mapping_epoch":15,"log_start":"155'551266","ondisk_log_start":"155'551266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7351275520,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":551044,"num_read_kb":2172436,"num_write":549416,"num_write_kb":2197664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9917,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:36.779305+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:57:36.779305+0000","last_peered":"2023-10-04T01:57:36.779305+0000","last_clean":"2023-10-04T01:57:36.779305+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:57:36.779305+0000","last_undegraded":"2023-10-04T01:57:36.779305+0000","last_fullsized":"2023-10-04T01:57:36.779305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'534633","reported_seq":1056636,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:57:39.540506+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:57:39.540506+0000","last_peered":"2023-10-04T01:57:39.540506+0000","last_clean":"2023-10-04T01:57:39.540506+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:57:39.540506+0000","last_undegraded":"2023-10-04T01:57:39.540506+0000","last_fullsized":"2023-10-04T01:57:39.540506+0000","mapping_epoch":15,"log_start":"155'524563","ondisk_log_start":"155'524563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2026,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 284s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7025262611,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":526866,"num_read_kb":2076276,"num_write":526482,"num_write_kb":2105922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'18955","reported_seq":24685,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:39.145036+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:57:39.145036+0000","last_peered":"2023-10-04T01:57:39.145036+0000","last_clean":"2023-10-04T01:57:39.145036+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:57:39.145036+0000","last_undegraded":"2023-10-04T01:57:39.145036+0000","last_fullsized":"2023-10-04T01:57:39.145036+0000","mapping_epoch":12,"log_start":"95'8900","ondisk_log_start":"95'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":335,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-04T01:55:43.925275+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:55:43.925275+0000","last_peered":"2023-10-04T01:55:43.925275+0000","last_clean":"2023-10-04T01:55:43.925275+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:55:43.925275+0000","last_undegraded":"2023-10-04T01:55:43.925275+0000","last_fullsized":"2023-10-04T01:55:43.925275+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28583227411,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2143778,"num_read_kb":8449220,"num_write":2140090,"num_write_kb":8560354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17530429440,"data_stored":37217525798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40162,"ondisk_log_size":40162,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":59884,"ondisk_log_size":59884,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739383,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10462884,"kb_used_data":8560988,"kb_used_omap":2628,"kb_used_meta":1899259,"kb_avail":83908956,"statfs":{"total":96636764160,"available":85922770944,"internally_reserved":0,"allocated":8766451712,"data_stored":18608686311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":1944842119},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:03 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.42799999999999999,"15min":0.46800000000000003},"min":{"1min":0.17100000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.71199999999999997,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":0.19},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.47699999999999998,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79000000000000004,"5min":4.492,"15min":6.7930000000000001},"last":0.25800000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:56:51 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.374,"15min":0.58199999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.74399999999999999,"5min":1.349,"15min":33.975000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.436,"15min":0.56599999999999995},"min":{"1min":0.23400000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.88,"5min":5.2240000000000002,"15min":35.423999999999999},"last":0.46800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739381,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5858548,"kb_used_data":4213932,"kb_used_omap":1191,"kb_used_meta":1643416,"kb_avail":88513292,"statfs":{"total":96636764160,"available":90637611008,"internally_reserved":0,"allocated":4315066368,"data_stored":9164798583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1682858361},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,4,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.314,"15min":0.33200000000000002},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.316},{"interface":"front","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.8460000000000001,"5min":6.431,"15min":6.7169999999999996},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.4219999999999997,"5min":4.641,"15min":10.494999999999999},"last":0.214},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.57899999999999996},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.3730000000000002,"5min":4.3730000000000002,"15min":10.531000000000001},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6112764,"kb_used_data":4351348,"kb_used_omap":1357,"kb_used_meta":1760050,"kb_avail":88259076,"statfs":{"total":96636764160,"available":90377293824,"internally_reserved":0,"allocated":4455780352,"data_stored":9446317518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1802292092},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:56:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.216,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":17.991},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.51300000000000001,"15min":0.621},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:56:59 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.80300000000000005,"15min":0.66100000000000003},"min":{"1min":0.26100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":5.4450000000000003,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.309},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.41199999999999998,"15min":0.46600000000000003},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":14.051},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4453666816,"data_stored":9445642240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4312158208,"data_stored":9163730963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8764604416,"data_stored":18608152595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:57:41.817 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:57:41.818 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:58:01.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:58:01.930 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:58:01.931 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:58:01.932 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:58:01.932 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:58:01.932 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:58:01.932 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:01.925+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:58:02.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:58:02.385 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:58:02.415 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2700,"stamp":"2023-10-04T01:58:01.819354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28616241721,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2363275,"num_read_kb":8659249,"num_write":2203615,"num_write_kb":8647999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100403,"ondisk_log_size":100403,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22741788,"kb_used_data":17188868,"kb_used_omap":5176,"kb_used_meta":5547719,"kb_avail":260373732,"statfs":{"total":289910292480,"available":266622701568,"internally_reserved":0,"allocated":17601400832,"data_stored":37283900669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5301124,"internal_metadata":5680864380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,4,7,16,16,18,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16269312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4184,"num_read_kb":16672,"num_write":3947,"num_write_kb":15788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001801"},"pg_stats":[{"pgid":"2.7","version":"156'8926","reported_seq":12542,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.259496+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:57:57.259496+0000","last_peered":"2023-10-04T01:57:57.259496+0000","last_clean":"2023-10-04T01:57:57.259496+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:57:57.259496+0000","last_undegraded":"2023-10-04T01:57:57.259496+0000","last_fullsized":"2023-10-04T01:57:57.259496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8926,"log_dups_size":0,"ondisk_log_size":8926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'7719","reported_seq":11534,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.258834+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:57:57.258834+0000","last_peered":"2023-10-04T01:57:57.258834+0000","last_clean":"2023-10-04T01:57:57.258834+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:57:57.258834+0000","last_undegraded":"2023-10-04T01:57:57.258834+0000","last_fullsized":"2023-10-04T01:57:57.258834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7719,"log_dups_size":0,"ondisk_log_size":7719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'9075","reported_seq":12999,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.258894+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:57:57.258894+0000","last_peered":"2023-10-04T01:57:57.258894+0000","last_clean":"2023-10-04T01:57:57.258894+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:57:57.258894+0000","last_undegraded":"2023-10-04T01:57:57.258894+0000","last_fullsized":"2023-10-04T01:57:57.258894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9075,"log_dups_size":0,"ondisk_log_size":9075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'7504","reported_seq":12274,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.259485+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:57:57.259485+0000","last_peered":"2023-10-04T01:57:57.259485+0000","last_clean":"2023-10-04T01:57:57.259485+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:57:57.259485+0000","last_undegraded":"2023-10-04T01:57:57.259485+0000","last_fullsized":"2023-10-04T01:57:57.259485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'551435","reported_seq":1087407,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.727304+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:57:57.727304+0000","last_peered":"2023-10-04T01:57:57.727304+0000","last_clean":"2023-10-04T01:57:57.727304+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:57:57.727304+0000","last_undegraded":"2023-10-04T01:57:57.727304+0000","last_fullsized":"2023-10-04T01:57:57.727304+0000","mapping_epoch":15,"log_start":"155'541373","ondisk_log_start":"155'541373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7186173952,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":540870,"num_read_kb":2131588,"num_write":539886,"num_write_kb":2159544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"156'18078","reported_seq":23593,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.259753+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:57:57.259753+0000","last_peered":"2023-10-04T01:57:57.259753+0000","last_clean":"2023-10-04T01:57:57.259753+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:57:57.259753+0000","last_undegraded":"2023-10-04T01:57:57.259753+0000","last_fullsized":"2023-10-04T01:57:57.259753+0000","mapping_epoch":12,"log_start":"95'8000","ondisk_log_start":"95'8000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'539616","reported_seq":1063942,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:57:59.683346+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:57:59.683346+0000","last_peered":"2023-10-04T01:57:59.683346+0000","last_clean":"2023-10-04T01:57:59.683346+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:57:59.683346+0000","last_undegraded":"2023-10-04T01:57:59.683346+0000","last_fullsized":"2023-10-04T01:57:59.683346+0000","mapping_epoch":15,"log_start":"155'529567","ondisk_log_start":"155'529567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7036682240,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":529143,"num_read_kb":2085444,"num_write":528236,"num_write_kb":2112944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9501,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.259073+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:57:57.259073+0000","last_peered":"2023-10-04T01:57:57.259073+0000","last_clean":"2023-10-04T01:57:57.259073+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:57:57.259073+0000","last_undegraded":"2023-10-04T01:57:57.259073+0000","last_fullsized":"2023-10-04T01:57:57.259073+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'563345","reported_seq":1111587,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.733176+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:57:57.733176+0000","last_peered":"2023-10-04T01:57:57.733176+0000","last_clean":"2023-10-04T01:57:57.733176+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:57:57.733176+0000","last_undegraded":"2023-10-04T01:57:57.733176+0000","last_fullsized":"2023-10-04T01:57:57.733176+0000","mapping_epoch":15,"log_start":"155'553266","ondisk_log_start":"155'553266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7359774720,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":553249,"num_read_kb":2181220,"num_write":551484,"num_write_kb":2205936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9930,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.268727+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:57:57.268727+0000","last_peered":"2023-10-04T01:57:57.268727+0000","last_clean":"2023-10-04T01:57:57.268727+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:57:57.268727+0000","last_undegraded":"2023-10-04T01:57:57.268727+0000","last_fullsized":"2023-10-04T01:57:57.268727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'536655","reported_seq":1060626,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:57:59.529900+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:57:59.529900+0000","last_peered":"2023-10-04T01:57:59.529900+0000","last_clean":"2023-10-04T01:57:59.529900+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:57:59.529900+0000","last_undegraded":"2023-10-04T01:57:59.529900+0000","last_fullsized":"2023-10-04T01:57:59.529900+0000","mapping_epoch":15,"log_start":"155'526563","ondisk_log_start":"155'526563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2195,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 304s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7033004051,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":528839,"num_read_kb":2084144,"num_write":528365,"num_write_kb":2113454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'18982","reported_seq":24725,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:57.259646+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:57:57.259646+0000","last_peered":"2023-10-04T01:57:57.259646+0000","last_clean":"2023-10-04T01:57:57.259646+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:57:57.259646+0000","last_undegraded":"2023-10-04T01:57:57.259646+0000","last_fullsized":"2023-10-04T01:57:57.259646+0000","mapping_epoch":12,"log_start":"95'8900","ondisk_log_start":"95'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":337,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:44.590292+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:44.590292+0000","last_peered":"2023-10-04T01:57:44.590292+0000","last_clean":"2023-10-04T01:57:44.590292+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:44.590292+0000","last_undegraded":"2023-10-04T01:57:44.590292+0000","last_fullsized":"2023-10-04T01:57:44.590292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28615634963,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2152101,"num_read_kb":8482396,"num_write":2147971,"num_write_kb":8591878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17594994688,"data_stored":37282091046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40282,"ondisk_log_size":40282,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60022,"ondisk_log_size":60022,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739387,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10630724,"kb_used_data":8592124,"kb_used_omap":2628,"kb_used_meta":2035963,"kb_avail":83741116,"statfs":{"total":96636764160,"available":85750902784,"internally_reserved":0,"allocated":8798334976,"data_stored":18640569575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2691193,"internal_metadata":2084827015},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:03 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.42799999999999999,"15min":0.46800000000000003},"min":{"1min":0.17100000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.71199999999999997,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":0.249},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.47699999999999998,"15min":0.52600000000000002},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79000000000000004,"5min":4.492,"15min":6.7930000000000001},"last":0.32900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:57:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.58799999999999997},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.165,"5min":3.165,"15min":33.975000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.55500000000000005},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":3.1019999999999999,"5min":3.706,"15min":35.423999999999999},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359739385,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5928012,"kb_used_data":4229252,"kb_used_omap":1191,"kb_used_meta":1697560,"kb_avail":88443828,"statfs":{"total":96636764160,"available":90566479872,"internally_reserved":0,"allocated":4330754048,"data_stored":9180482120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1738301817},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,2,4,6,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.314,"15min":0.33200000000000002},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":3.1869999999999998},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.8460000000000001,"5min":6.431,"15min":6.7169999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.183,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.4219999999999997,"5min":4.641,"15min":10.494999999999999},"last":0.255},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.57899999999999996},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.3730000000000002,"5min":4.3730000000000002,"15min":10.531000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6183052,"kb_used_data":4367492,"kb_used_omap":1357,"kb_used_meta":1814194,"kb_avail":88188788,"statfs":{"total":96636764160,"available":90305318912,"internally_reserved":0,"allocated":4472311808,"data_stored":9462848974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1857735548},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,9,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:57:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46400000000000002,"15min":0.50800000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.64500000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.50900000000000001,"15min":0.626},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21099999999999999},"max":{"1min":1.9930000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Oct 4 01:56:59 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.80300000000000005,"15min":0.66100000000000003},"min":{"1min":0.26100000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":5.4450000000000003,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.41199999999999998,"15min":0.46600000000000003},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":14.051},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4470255616,"data_stored":9462231040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4327833600,"data_stored":9179406355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8796905472,"data_stored":18640453651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:58:02.417 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:58:02.417 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:58:17.616 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:58:17.608+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T01:58:17.617 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T01:58:17.608+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T01:58:19.800 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:58:19.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:58:19.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:58:19.804 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:58:19.804 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:58:19.804 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:58:19.804 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:58:19.805 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:58:19.805 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:58:19.805 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:19.792+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:58:22.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:58:22.942 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:58:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:58:22.990 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2710,"stamp":"2023-10-04T01:58:21.823675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28648436281,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2371233,"num_read_kb":8690997,"num_write":2211457,"num_write_kb":8679367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100314,"ondisk_log_size":100314,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22070116,"kb_used_data":17250952,"kb_used_omap":5326,"kb_used_meta":4813809,"kb_avail":261045404,"statfs":{"total":289910292480,"available":267310493696,"internally_reserved":0,"allocated":17664974848,"data_stored":37347466399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5453998,"internal_metadata":4929341266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,7,5,12,9,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15970304,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4029,"num_read_kb":16076,"num_write":3891,"num_write_kb":15564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002001"},"pg_stats":[{"pgid":"2.7","version":"157'8967","reported_seq":12592,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.777119+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:58:19.777119+0000","last_peered":"2023-10-04T01:58:19.777119+0000","last_clean":"2023-10-04T01:58:19.777119+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:58:19.777119+0000","last_undegraded":"2023-10-04T01:58:19.777119+0000","last_fullsized":"2023-10-04T01:58:19.777119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8967,"log_dups_size":0,"ondisk_log_size":8967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'7740","reported_seq":11560,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.518348+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:58:19.518348+0000","last_peered":"2023-10-04T01:58:19.518348+0000","last_clean":"2023-10-04T01:58:19.518348+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:58:19.518348+0000","last_undegraded":"2023-10-04T01:58:19.518348+0000","last_fullsized":"2023-10-04T01:58:19.518348+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'9116","reported_seq":13048,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.776973+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:58:19.776973+0000","last_peered":"2023-10-04T01:58:19.776973+0000","last_clean":"2023-10-04T01:58:19.776973+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:58:19.776973+0000","last_undegraded":"2023-10-04T01:58:19.776973+0000","last_fullsized":"2023-10-04T01:58:19.776973+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'7546","reported_seq":12324,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.777053+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:58:19.777053+0000","last_peered":"2023-10-04T01:58:19.777053+0000","last_clean":"2023-10-04T01:58:19.777053+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:58:19.777053+0000","last_undegraded":"2023-10-04T01:58:19.777053+0000","last_fullsized":"2023-10-04T01:58:19.777053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7546,"log_dups_size":0,"ondisk_log_size":7546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'553418","reported_seq":1091418,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:17.831838+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:58:17.831838+0000","last_peered":"2023-10-04T01:58:17.831838+0000","last_clean":"2023-10-04T01:58:17.831838+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:58:17.831838+0000","last_undegraded":"2023-10-04T01:58:17.831838+0000","last_fullsized":"2023-10-04T01:58:17.831838+0000","mapping_epoch":15,"log_start":"155'543373","ondisk_log_start":"155'543373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7194365952,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":542896,"num_read_kb":2139668,"num_write":541869,"num_write_kb":2167476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'18126","reported_seq":23653,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:17.823265+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:58:17.823265+0000","last_peered":"2023-10-04T01:58:17.823265+0000","last_clean":"2023-10-04T01:58:17.823265+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:58:17.823265+0000","last_undegraded":"2023-10-04T01:58:17.823265+0000","last_fullsized":"2023-10-04T01:58:17.823265+0000","mapping_epoch":12,"log_start":"95'8100","ondisk_log_start":"95'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'541524","reported_seq":1067846,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.738048+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:58:19.738048+0000","last_peered":"2023-10-04T01:58:19.738048+0000","last_clean":"2023-10-04T01:58:19.738048+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:58:19.738048+0000","last_undegraded":"2023-10-04T01:58:19.738048+0000","last_fullsized":"2023-10-04T01:58:19.738048+0000","mapping_epoch":15,"log_start":"155'531467","ondisk_log_start":"155'531467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7044505600,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":531141,"num_read_kb":2093420,"num_write":530144,"num_write_kb":2120576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9518,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:17.739546+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:58:17.739546+0000","last_peered":"2023-10-04T01:58:17.739546+0000","last_clean":"2023-10-04T01:58:17.739546+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:58:17.739546+0000","last_undegraded":"2023-10-04T01:58:17.739546+0000","last_fullsized":"2023-10-04T01:58:17.739546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'565375","reported_seq":1115623,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:17.827451+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:58:17.827451+0000","last_peered":"2023-10-04T01:58:17.827451+0000","last_clean":"2023-10-04T01:58:17.827451+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:58:17.827451+0000","last_undegraded":"2023-10-04T01:58:17.827451+0000","last_fullsized":"2023-10-04T01:58:17.827451+0000","mapping_epoch":15,"log_start":"156'555366","ondisk_log_start":"156'555366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7368081408,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":555255,"num_read_kb":2189208,"num_write":553514,"num_write_kb":2214056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9943,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:17.738842+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:58:17.738842+0000","last_peered":"2023-10-04T01:58:17.738842+0000","last_clean":"2023-10-04T01:58:17.738842+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:58:17.738842+0000","last_undegraded":"2023-10-04T01:58:17.738842+0000","last_fullsized":"2023-10-04T01:58:17.738842+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'538693","reported_seq":1064624,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:58:19.780969+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:58:19.780969+0000","last_peered":"2023-10-04T01:58:19.780969+0000","last_clean":"2023-10-04T01:58:19.780969+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:58:19.780969+0000","last_undegraded":"2023-10-04T01:58:19.780969+0000","last_fullsized":"2023-10-04T01:58:19.780969+0000","mapping_epoch":15,"log_start":"156'528663","ondisk_log_start":"156'528663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2340,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 324s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7040876563,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":530767,"num_read_kb":2091848,"num_write":530286,"num_write_kb":2121138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'19041","reported_seq":24796,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:19.571570+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:58:19.571570+0000","last_peered":"2023-10-04T01:58:19.571570+0000","last_clean":"2023-10-04T01:58:19.571570+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:58:19.571570+0000","last_undegraded":"2023-10-04T01:58:19.571570+0000","last_fullsized":"2023-10-04T01:58:19.571570+0000","mapping_epoch":12,"log_start":"95'9000","ondisk_log_start":"95'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":337,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:44.590292+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:44.590292+0000","last_peered":"2023-10-04T01:57:44.590292+0000","last_clean":"2023-10-04T01:57:44.590292+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:44.590292+0000","last_undegraded":"2023-10-04T01:57:44.590292+0000","last_fullsized":"2023-10-04T01:57:44.590292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28647829523,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2160059,"num_read_kb":8514144,"num_write":2155813,"num_write_kb":8623246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17659244544,"data_stored":37346340902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60074,"ondisk_log_size":60074,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739391,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9819888,"kb_used_data":8623332,"kb_used_omap":2777,"kb_used_meta":1193766,"kb_avail":84551952,"statfs":{"total":96636764160,"available":86581198848,"internally_reserved":0,"allocated":8830291968,"data_stored":18672522424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1222416989},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.42099999999999999,"15min":0.46100000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.67300000000000004,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.70399999999999996,"5min":4.492,"15min":6.7930000000000001},"last":0.34799999999999998}]},{"osd":1,"last update":"Wed Oct 4 01:57:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.58799999999999997},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.165,"5min":3.165,"15min":33.975000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.55500000000000005},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":3.1019999999999999,"5min":3.706,"15min":35.423999999999999},"last":0.24399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739389,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5979152,"kb_used_data":4244296,"kb_used_omap":1191,"kb_used_meta":1733656,"kb_avail":88392688,"statfs":{"total":96636764160,"available":90514112512,"internally_reserved":0,"allocated":4346159104,"data_stored":9195887176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1775264121},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,5,6,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.307,"15min":0.32100000000000001},"min":{"1min":0.22900000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.45500000000000002,"5min":0.91200000000000003,"15min":3.1850000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.49099999999999999,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.1339999999999999,"5min":4.2699999999999996,"15min":6.7169999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.21199999999999999,"5min":0.18099999999999999,"15min":0.17299999999999999},"max":{"1min":3.4159999999999999,"5min":4.641,"15min":10.494999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.21199999999999999},"max":{"1min":1.3220000000000001,"5min":4.3730000000000002,"15min":10.531000000000001},"last":0.35099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6271076,"kb_used_data":4383324,"kb_used_omap":1357,"kb_used_meta":1886386,"kb_avail":88100764,"statfs":{"total":96636764160,"available":90215182336,"internally_reserved":0,"allocated":4488523776,"data_stored":9479056799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1931660156},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:57:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46400000000000002,"15min":0.50800000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.64500000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.50900000000000001,"15min":0.626},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21099999999999999},"max":{"1min":1.9930000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Oct 4 01:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.79700000000000004,"15min":0.65800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.70999999999999996,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.246},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.42099999999999999,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.76100000000000001,"5min":5.2309999999999999,"15min":14.051},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4486692864,"data_stored":9478668288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4343525376,"data_stored":9195098131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8829026304,"data_stored":18672574483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:58:22.992 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:58:22.993 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:58:42.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:58:43.659 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:58:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:58:43.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2720,"stamp":"2023-10-04T01:58:41.829269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28680536633,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2379334,"num_read_kb":8723293,"num_write":2219264,"num_write_kb":8710595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100580,"ondisk_log_size":100580,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22313516,"kb_used_data":17313940,"kb_used_omap":5326,"kb_used_meta":4994225,"kb_avail":260802004,"statfs":{"total":289910292480,"available":267061252096,"internally_reserved":0,"allocated":17729474560,"data_stored":37411966111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5453998,"internal_metadata":5114087250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,4,7,7,17,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19709952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5036,"num_read_kb":20096,"num_write":4797,"num_write_kb":19188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004237"},"pg_stats":[{"pgid":"2.7","version":"157'8987","reported_seq":12620,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:39.655877+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:58:39.655877+0000","last_peered":"2023-10-04T01:58:39.655877+0000","last_clean":"2023-10-04T01:58:39.655877+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:58:39.655877+0000","last_undegraded":"2023-10-04T01:58:39.655877+0000","last_fullsized":"2023-10-04T01:58:39.655877+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'7750","reported_seq":11574,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:38.219320+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:58:38.219320+0000","last_peered":"2023-10-04T01:58:38.219320+0000","last_clean":"2023-10-04T01:58:38.219320+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:58:38.219320+0000","last_undegraded":"2023-10-04T01:58:38.219320+0000","last_fullsized":"2023-10-04T01:58:38.219320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'9136","reported_seq":13076,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:39.655830+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:58:39.655830+0000","last_peered":"2023-10-04T01:58:39.655830+0000","last_clean":"2023-10-04T01:58:39.655830+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:58:39.655830+0000","last_undegraded":"2023-10-04T01:58:39.655830+0000","last_fullsized":"2023-10-04T01:58:39.655830+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'7566","reported_seq":12352,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:39.651324+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:58:39.651324+0000","last_peered":"2023-10-04T01:58:39.651324+0000","last_clean":"2023-10-04T01:58:39.651324+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:58:39.651324+0000","last_undegraded":"2023-10-04T01:58:39.651324+0000","last_fullsized":"2023-10-04T01:58:39.651324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'555464","reported_seq":1095494,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:37.969498+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:58:37.969498+0000","last_peered":"2023-10-04T01:58:37.969498+0000","last_clean":"2023-10-04T01:58:37.969498+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:58:37.969498+0000","last_undegraded":"2023-10-04T01:58:37.969498+0000","last_fullsized":"2023-10-04T01:58:37.969498+0000","mapping_epoch":15,"log_start":"156'545373","ondisk_log_start":"156'545373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7202803712,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":544927,"num_read_kb":2147772,"num_write":543915,"num_write_kb":2175660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'18144","reported_seq":23681,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:37.395032+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:58:37.395032+0000","last_peered":"2023-10-04T01:58:37.395032+0000","last_clean":"2023-10-04T01:58:37.395032+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:58:37.395032+0000","last_undegraded":"2023-10-04T01:58:37.395032+0000","last_fullsized":"2023-10-04T01:58:37.395032+0000","mapping_epoch":12,"log_start":"95'8100","ondisk_log_start":"95'8100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'543406","reported_seq":1071797,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:39.903804+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:58:39.903804+0000","last_peered":"2023-10-04T01:58:39.903804+0000","last_clean":"2023-10-04T01:58:39.903804+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:58:39.903804+0000","last_undegraded":"2023-10-04T01:58:39.903804+0000","last_fullsized":"2023-10-04T01:58:39.903804+0000","mapping_epoch":15,"log_start":"156'533367","ondisk_log_start":"156'533367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7052251136,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":533210,"num_read_kb":2101680,"num_write":532026,"num_write_kb":2128104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9530,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:33.099685+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:58:33.099685+0000","last_peered":"2023-10-04T01:58:33.099685+0000","last_clean":"2023-10-04T01:58:33.099685+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:58:33.099685+0000","last_undegraded":"2023-10-04T01:58:33.099685+0000","last_fullsized":"2023-10-04T01:58:33.099685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'567360","reported_seq":1119611,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:37.957798+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:58:37.957798+0000","last_peered":"2023-10-04T01:58:37.957798+0000","last_clean":"2023-10-04T01:58:37.957798+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:58:37.957798+0000","last_undegraded":"2023-10-04T01:58:37.957798+0000","last_fullsized":"2023-10-04T01:58:37.957798+0000","mapping_epoch":15,"log_start":"156'557266","ondisk_log_start":"156'557266","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7376248832,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":557259,"num_read_kb":2197172,"num_write":555499,"num_write_kb":2221996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9952,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:33.103667+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:58:33.103667+0000","last_peered":"2023-10-04T01:58:33.103667+0000","last_clean":"2023-10-04T01:58:33.103667+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:58:33.103667+0000","last_undegraded":"2023-10-04T01:58:33.103667+0000","last_fullsized":"2023-10-04T01:58:33.103667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'540725","reported_seq":1068664,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:58:39.895314+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:58:39.895314+0000","last_peered":"2023-10-04T01:58:39.895314+0000","last_clean":"2023-10-04T01:58:39.895314+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:58:39.895314+0000","last_undegraded":"2023-10-04T01:58:39.895314+0000","last_fullsized":"2023-10-04T01:58:39.895314+0000","mapping_epoch":15,"log_start":"156'530663","ondisk_log_start":"156'530663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2508,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 344s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7048626195,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":532764,"num_read_kb":2099816,"num_write":532180,"num_write_kb":2128714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'19074","reported_seq":24841,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:39.672363+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:58:39.672363+0000","last_peered":"2023-10-04T01:58:39.672363+0000","last_clean":"2023-10-04T01:58:39.672363+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:58:39.672363+0000","last_undegraded":"2023-10-04T01:58:39.672363+0000","last_fullsized":"2023-10-04T01:58:39.672363+0000","mapping_epoch":12,"log_start":"95'9000","ondisk_log_start":"95'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":337,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:44.590292+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:44.590292+0000","last_peered":"2023-10-04T01:57:44.590292+0000","last_clean":"2023-10-04T01:57:44.590292+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:44.590292+0000","last_undegraded":"2023-10-04T01:57:44.590292+0000","last_fullsized":"2023-10-04T01:57:44.590292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28679929875,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2168160,"num_read_kb":8546440,"num_write":2163620,"num_write_kb":8654474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17723060224,"data_stored":37410156582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60195,"ondisk_log_size":60195,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739395,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9918852,"kb_used_data":8654268,"kb_used_omap":2777,"kb_used_meta":1261798,"kb_avail":84452988,"statfs":{"total":96636764160,"available":86479859712,"internally_reserved":0,"allocated":8861970432,"data_stored":18704200888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1292081757},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.42099999999999999,"15min":0.46100000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.67300000000000004,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":4.665},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.70399999999999996,"5min":4.492,"15min":6.7930000000000001},"last":4.7569999999999997}]},{"osd":1,"last update":"Wed Oct 4 01:57:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.58799999999999997},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.165,"5min":3.165,"15min":33.975000000000001},"last":4.7210000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.55500000000000005},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":3.1019999999999999,"5min":3.706,"15min":35.423999999999999},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739393,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6066464,"kb_used_data":4259480,"kb_used_omap":1191,"kb_used_meta":1805784,"kb_avail":88305376,"statfs":{"total":96636764160,"available":90424705024,"internally_reserved":0,"allocated":4361707520,"data_stored":9211435592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1849123193},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,3,2,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.307,"15min":0.32100000000000001},"min":{"1min":0.22900000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.45500000000000002,"5min":0.91200000000000003,"15min":3.1850000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.49099999999999999,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.1339999999999999,"5min":4.2699999999999996,"15min":6.7169999999999996},"last":0.308}]},{"osd":2,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.21199999999999999,"5min":0.18099999999999999,"15min":0.17299999999999999},"max":{"1min":3.4159999999999999,"5min":4.641,"15min":10.494999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.21199999999999999},"max":{"1min":1.3220000000000001,"5min":4.3730000000000002,"15min":10.531000000000001},"last":0.38800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6328200,"kb_used_data":4400192,"kb_used_omap":1357,"kb_used_meta":1926642,"kb_avail":88043640,"statfs":{"total":96636764160,"available":90156687360,"internally_reserved":0,"allocated":4505796608,"data_stored":9496329631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":1972882300},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:57:48 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46400000000000002,"15min":0.50800000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.188},"max":{"1min":0.64500000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.50900000000000001,"15min":0.626},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.21099999999999999},"max":{"1min":1.9930000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.79700000000000004,"15min":0.65800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.70999999999999996,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.42099999999999999,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.76100000000000001,"5min":5.2309999999999999,"15min":14.051},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4503207936,"data_stored":9495183360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4358983680,"data_stored":9210556435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8860868608,"data_stored":18704416787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:58:43.689 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:58:43.691 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:58:53.649 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.636+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:58:53.652 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.636+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:58:53.652 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.636+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:58:53.652 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.640+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:58:53.652 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.640+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:58:53.652 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:58:53.640+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T01:59:03.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:59:04.266 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:59:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:59:04.322 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2732,"stamp":"2023-10-04T01:59:03.834261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28717486649,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2388563,"num_read_kb":8760085,"num_write":2228246,"num_write_kb":8746523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100502,"ondisk_log_size":100502,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22652672,"kb_used_data":17389352,"kb_used_omap":5326,"kb_used_meta":5257969,"kb_avail":260462848,"statfs":{"total":289910292480,"available":266713956352,"internally_reserved":0,"allocated":17806696448,"data_stored":37489187999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5453998,"internal_metadata":5384161106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,11,9,13,19,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16343040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4144,"num_read_kb":16524,"num_write":3959,"num_write_kb":15836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003071"},"pg_stats":[{"pgid":"2.7","version":"157'9017","reported_seq":12658,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.700106+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:58:58.700106+0000","last_peered":"2023-10-04T01:58:58.700106+0000","last_clean":"2023-10-04T01:58:58.700106+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:58:58.700106+0000","last_undegraded":"2023-10-04T01:58:58.700106+0000","last_fullsized":"2023-10-04T01:58:58.700106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9017,"log_dups_size":0,"ondisk_log_size":9017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'7765","reported_seq":11593,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:55.664120+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:58:55.664120+0000","last_peered":"2023-10-04T01:58:55.664120+0000","last_clean":"2023-10-04T01:58:55.664120+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:58:55.664120+0000","last_undegraded":"2023-10-04T01:58:55.664120+0000","last_fullsized":"2023-10-04T01:58:55.664120+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'9166","reported_seq":13114,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.699720+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:58:58.699720+0000","last_peered":"2023-10-04T01:58:58.699720+0000","last_clean":"2023-10-04T01:58:58.699720+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:58:58.699720+0000","last_undegraded":"2023-10-04T01:58:58.699720+0000","last_fullsized":"2023-10-04T01:58:58.699720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'7596","reported_seq":12390,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.699984+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:58:58.699984+0000","last_peered":"2023-10-04T01:58:58.699984+0000","last_clean":"2023-10-04T01:58:58.699984+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:58:58.699984+0000","last_undegraded":"2023-10-04T01:58:58.699984+0000","last_fullsized":"2023-10-04T01:58:58.699984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'558001","reported_seq":1100664,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:59:03.031401+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:59:03.031401+0000","last_peered":"2023-10-04T01:59:03.031401+0000","last_clean":"2023-10-04T01:59:03.031401+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:59:03.031401+0000","last_undegraded":"2023-10-04T01:59:03.031401+0000","last_fullsized":"2023-10-04T01:59:03.031401+0000","mapping_epoch":15,"log_start":"156'547973","ondisk_log_start":"156'547973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7213178880,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":547559,"num_read_kb":2158268,"num_write":546452,"num_write_kb":2185808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'18213","reported_seq":23765,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:59:01.530994+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:59:01.530994+0000","last_peered":"2023-10-04T01:59:01.530994+0000","last_clean":"2023-10-04T01:59:01.530994+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:59:01.530994+0000","last_undegraded":"2023-10-04T01:59:01.530994+0000","last_fullsized":"2023-10-04T01:59:01.530994+0000","mapping_epoch":12,"log_start":"95'8200","ondisk_log_start":"95'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'545399","reported_seq":1075898,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:59:00.029200+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:59:00.029200+0000","last_peered":"2023-10-04T01:59:00.029200+0000","last_clean":"2023-10-04T01:59:00.029200+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:59:00.029200+0000","last_undegraded":"2023-10-04T01:59:00.029200+0000","last_fullsized":"2023-10-04T01:59:00.029200+0000","mapping_epoch":15,"log_start":"156'535367","ondisk_log_start":"156'535367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7060484096,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":535316,"num_read_kb":2110072,"num_write":534019,"num_write_kb":2136076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9551,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.700315+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:58:58.700315+0000","last_peered":"2023-10-04T01:58:58.700315+0000","last_clean":"2023-10-04T01:58:58.700315+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:58:58.700315+0000","last_undegraded":"2023-10-04T01:58:58.700315+0000","last_fullsized":"2023-10-04T01:58:58.700315+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'569941","reported_seq":1124723,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:59:03.026527+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:59:03.026527+0000","last_peered":"2023-10-04T01:59:03.026527+0000","last_clean":"2023-10-04T01:59:03.026527+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:59:03.026527+0000","last_undegraded":"2023-10-04T01:59:03.026527+0000","last_fullsized":"2023-10-04T01:59:03.026527+0000","mapping_epoch":15,"log_start":"156'559866","ondisk_log_start":"156'559866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7386861568,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":559791,"num_read_kb":2207264,"num_write":558080,"num_write_kb":2232320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9967,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.698967+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:58:58.698967+0000","last_peered":"2023-10-04T01:58:58.698967+0000","last_clean":"2023-10-04T01:58:58.698967+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:58:58.698967+0000","last_undegraded":"2023-10-04T01:58:58.698967+0000","last_fullsized":"2023-10-04T01:58:58.698967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'542717","reported_seq":1072627,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:59:00.053401+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:59:00.053401+0000","last_peered":"2023-10-04T01:59:00.053401+0000","last_clean":"2023-10-04T01:59:00.053401+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:59:00.053401+0000","last_undegraded":"2023-10-04T01:59:00.053401+0000","last_fullsized":"2023-10-04T01:59:00.053401+0000","mapping_epoch":15,"log_start":"156'532663","ondisk_log_start":"156'532663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2657,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 364s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7056355347,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":534723,"num_read_kb":2107628,"num_write":534051,"num_write_kb":2136198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'19119","reported_seq":24898,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:58:58.700341+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:58:58.700341+0000","last_peered":"2023-10-04T01:58:58.700341+0000","last_clean":"2023-10-04T01:58:58.700341+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:58:58.700341+0000","last_undegraded":"2023-10-04T01:58:58.700341+0000","last_fullsized":"2023-10-04T01:58:58.700341+0000","mapping_epoch":12,"log_start":"95'9100","ondisk_log_start":"95'9100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":337,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-04T01:57:44.590292+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:44.590292+0000","last_peered":"2023-10-04T01:57:44.590292+0000","last_clean":"2023-10-04T01:57:44.590292+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:44.590292+0000","last_undegraded":"2023-10-04T01:57:44.590292+0000","last_fullsized":"2023-10-04T01:57:44.590292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28716879891,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2177389,"num_read_kb":8583232,"num_write":2172602,"num_write_kb":8690402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17800392704,"data_stored":37487489062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60214,"ondisk_log_size":60214,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10085048,"kb_used_data":8693808,"kb_used_omap":2777,"kb_used_meta":1388454,"kb_avail":84286792,"statfs":{"total":96636764160,"available":86309675008,"internally_reserved":0,"allocated":8902459392,"data_stored":18744689848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1421777501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.42099999999999999,"15min":0.46100000000000002},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.67300000000000004,"5min":4.4580000000000002,"15min":6.7199999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.47099999999999997,"15min":0.51900000000000002},"min":{"1min":0.25800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.70399999999999996,"5min":4.492,"15min":6.7930000000000001},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Oct 4 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.44500000000000001,"15min":0.60199999999999998},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.437,"15min":0.55600000000000005},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":4.0659999999999998,"5min":4.0659999999999998,"15min":35.423999999999999},"last":0.45600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739397,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6117936,"kb_used_data":4274920,"kb_used_omap":1191,"kb_used_meta":1841816,"kb_avail":88253904,"statfs":{"total":96636764160,"available":90371997696,"internally_reserved":0,"allocated":4377518080,"data_stored":9227246152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1886019961},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3,10,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.307,"15min":0.32100000000000001},"min":{"1min":0.22900000000000001,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.45500000000000002,"5min":0.91200000000000003,"15min":3.1850000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.49099999999999999,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.1339999999999999,"5min":4.2699999999999996,"15min":6.7169999999999996},"last":5.8440000000000003}]},{"osd":2,"last update":"Wed Oct 4 01:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.21199999999999999,"5min":0.18099999999999999,"15min":0.17299999999999999},"max":{"1min":3.4159999999999999,"5min":4.641,"15min":10.494999999999999},"last":2.9809999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.21199999999999999},"max":{"1min":1.3220000000000001,"5min":4.3730000000000002,"15min":10.531000000000001},"last":4.7519999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6449688,"kb_used_data":4420624,"kb_used_omap":1357,"kb_used_meta":2027698,"kb_avail":87922152,"statfs":{"total":96636764160,"available":90032283648,"internally_reserved":0,"allocated":4526718976,"data_stored":9517251999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389700,"internal_metadata":2076363644},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,4,10,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41799999999999998,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51800000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46899999999999997,"15min":0.60099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":0.69399999999999995,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Oct 4 01:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.79700000000000004,"15min":0.65800000000000003},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.70999999999999996,"5min":26.321999999999999,"15min":26.321999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.42099999999999999,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.20200000000000001},"max":{"1min":0.76100000000000001,"5min":5.2309999999999999,"15min":14.051},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4524183552,"data_stored":9516158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4374818816,"data_stored":9226391571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8901390336,"data_stored":18744938515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:59:04.327 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:59:04.327 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:59:24.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:59:24.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:59:24.798 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:59:24.842 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2742,"stamp":"2023-10-04T01:59:23.839109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28749865529,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2396611,"num_read_kb":8792173,"num_write":2236101,"num_write_kb":8777943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100736,"ondisk_log_size":100736,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22314144,"kb_used_data":17451912,"kb_used_omap":5327,"kb_used_meta":4856880,"kb_avail":260801376,"statfs":{"total":289910292480,"available":267060609024,"internally_reserved":0,"allocated":17870757888,"data_stored":37553236881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5455678,"internal_metadata":4973445314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,2,3,8,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20439040,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5013,"num_read_kb":19996,"num_write":4949,"num_write_kb":19796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002122"},"pg_stats":[{"pgid":"2.7","version":"158'9041","reported_seq":12691,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.842879+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:59:19.842879+0000","last_peered":"2023-10-04T01:59:19.842879+0000","last_clean":"2023-10-04T01:59:19.842879+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:59:19.842879+0000","last_undegraded":"2023-10-04T01:59:19.842879+0000","last_fullsized":"2023-10-04T01:59:19.842879+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'7777","reported_seq":11610,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.570685+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:59:19.570685+0000","last_peered":"2023-10-04T01:59:19.570685+0000","last_clean":"2023-10-04T01:59:19.570685+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:59:19.570685+0000","last_undegraded":"2023-10-04T01:59:19.570685+0000","last_fullsized":"2023-10-04T01:59:19.570685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'9190","reported_seq":13148,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.844188+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:59:19.844188+0000","last_peered":"2023-10-04T01:59:19.844188+0000","last_clean":"2023-10-04T01:59:19.844188+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:59:19.844188+0000","last_undegraded":"2023-10-04T01:59:19.844188+0000","last_fullsized":"2023-10-04T01:59:19.844188+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'7620","reported_seq":12424,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.839526+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:59:19.839526+0000","last_peered":"2023-10-04T01:59:19.839526+0000","last_clean":"2023-10-04T01:59:19.839526+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:59:19.839526+0000","last_undegraded":"2023-10-04T01:59:19.839526+0000","last_fullsized":"2023-10-04T01:59:19.839526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7620,"log_dups_size":0,"ondisk_log_size":7620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'559914","reported_seq":1104537,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:23.073893+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:59:23.073893+0000","last_peered":"2023-10-04T01:59:23.073893+0000","last_clean":"2023-10-04T01:59:23.073893+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:59:23.073893+0000","last_undegraded":"2023-10-04T01:59:23.073893+0000","last_fullsized":"2023-10-04T01:59:23.073893+0000","mapping_epoch":15,"log_start":"156'549873","ondisk_log_start":"156'549873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7221014528,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":549518,"num_read_kb":2166080,"num_write":548365,"num_write_kb":2193460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'18262","reported_seq":23828,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:22.847738+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:59:22.847738+0000","last_peered":"2023-10-04T01:59:22.847738+0000","last_clean":"2023-10-04T01:59:22.847738+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:59:22.847738+0000","last_undegraded":"2023-10-04T01:59:22.847738+0000","last_fullsized":"2023-10-04T01:59:22.847738+0000","mapping_epoch":12,"log_start":"95'8200","ondisk_log_start":"95'8200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'547357","reported_seq":1079800,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:20.120230+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:59:20.120230+0000","last_peered":"2023-10-04T01:59:20.120230+0000","last_clean":"2023-10-04T01:59:20.120230+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:59:20.120230+0000","last_undegraded":"2023-10-04T01:59:20.120230+0000","last_fullsized":"2023-10-04T01:59:20.120230+0000","mapping_epoch":15,"log_start":"156'537267","ondisk_log_start":"156'537267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7068553216,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":537258,"num_read_kb":2117824,"num_write":535977,"num_write_kb":2143908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9568,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.233474+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:59:19.233474+0000","last_peered":"2023-10-04T01:59:19.233474+0000","last_clean":"2023-10-04T01:59:19.233474+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:59:19.233474+0000","last_undegraded":"2023-10-04T01:59:19.233474+0000","last_fullsized":"2023-10-04T01:59:19.233474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'571914","reported_seq":1128800,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:23.075134+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:59:23.075134+0000","last_peered":"2023-10-04T01:59:23.075134+0000","last_clean":"2023-10-04T01:59:23.075134+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:59:23.075134+0000","last_undegraded":"2023-10-04T01:59:23.075134+0000","last_fullsized":"2023-10-04T01:59:23.075134+0000","mapping_epoch":15,"log_start":"156'561866","ondisk_log_start":"156'561866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7395000320,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":561891,"num_read_kb":2215644,"num_write":560053,"num_write_kb":2240212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9980,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.184472+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:59:19.184472+0000","last_peered":"2023-10-04T01:59:19.184472+0000","last_clean":"2023-10-04T01:59:19.184472+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:59:19.184472+0000","last_undegraded":"2023-10-04T01:59:19.184472+0000","last_fullsized":"2023-10-04T01:59:19.184472+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'544838","reported_seq":1076802,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:59:20.127445+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:59:20.127445+0000","last_peered":"2023-10-04T01:59:20.127445+0000","last_clean":"2023-10-04T01:59:20.127445+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:59:20.127445+0000","last_undegraded":"2023-10-04T01:59:20.127445+0000","last_fullsized":"2023-10-04T01:59:20.127445+0000","mapping_epoch":15,"log_start":"156'534763","ondisk_log_start":"156'534763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2794,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 384s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7064690707,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":536770,"num_read_kb":2115772,"num_write":536062,"num_write_kb":2144242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'19155","reported_seq":24948,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:19.857605+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:59:19.857605+0000","last_peered":"2023-10-04T01:59:19.857605+0000","last_clean":"2023-10-04T01:59:19.857605+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:59:19.857605+0000","last_undegraded":"2023-10-04T01:59:19.857605+0000","last_fullsized":"2023-10-04T01:59:19.857605+0000","mapping_epoch":12,"log_start":"95'9100","ondisk_log_start":"95'9100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":338,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:57:58.501390+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:58.501390+0000","last_peered":"2023-10-04T01:57:58.501390+0000","last_clean":"2023-10-04T01:57:58.501390+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:58.501390+0000","last_undegraded":"2023-10-04T01:57:58.501390+0000","last_fullsized":"2023-10-04T01:57:58.501390+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28749258771,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2185437,"num_read_kb":8615320,"num_write":2180457,"num_write_kb":8721822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17864744960,"data_stored":37551841318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60383,"ondisk_log_size":60383,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739404,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10202440,"kb_used_data":8725120,"kb_used_omap":2777,"kb_used_meta":1474534,"kb_avail":84169400,"statfs":{"total":96636764160,"available":86189465600,"internally_reserved":0,"allocated":8934522880,"data_stored":18776749150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1509923421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.47099999999999997,"15min":0.47899999999999998},"min":{"1min":0.192,"5min":0.158,"15min":0.158},"max":{"1min":4.665,"5min":4.665,"15min":6.7199999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.26200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.318}]},{"osd":1,"last update":"Wed Oct 4 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.44500000000000001,"15min":0.60199999999999998},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.437,"15min":0.55600000000000005},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":4.0659999999999998,"5min":4.0659999999999998,"15min":35.423999999999999},"last":0.29799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739401,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6187848,"kb_used_data":4290688,"kb_used_omap":1191,"kb_used_meta":1895960,"kb_avail":88183992,"statfs":{"total":96636764160,"available":90300407808,"internally_reserved":0,"allocated":4393664512,"data_stored":9243388398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":1941463417},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,8,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.309,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.65200000000000002,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.51200000000000001,"15min":0.54500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.8440000000000003,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.60299999999999998,"15min":0.51900000000000002},"min":{"1min":0.221,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9809999999999999,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.625,"15min":0.53700000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.34000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5923856,"kb_used_data":4436104,"kb_used_omap":1358,"kb_used_meta":1486385,"kb_avail":88447984,"statfs":{"total":96636764160,"available":90570735616,"internally_reserved":0,"allocated":4542570496,"data_stored":9533099333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1522058476},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41799999999999998,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51800000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46899999999999997,"15min":0.60099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":0.69399999999999995,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.46500000000000002,"15min":0.62},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":0.67200000000000004,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.6340000000000003,"5min":5.2309999999999999,"15min":14.051},"last":0.34499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4540080128,"data_stored":9532055552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4391071744,"data_stored":9242644499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8933593088,"data_stored":18777141267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:59:24.844 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:59:24.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T01:59:25.314 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T01:59:25.350 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:59:36.982 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:36.970+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:59:36.983 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:36.974+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:59:36.984 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:36.974+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:59:36.984 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:36.974+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:59:36.984 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:36.974+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:59:45.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T01:59:45.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T01:59:45.788 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T01:59:45.824 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2752,"stamp":"2023-10-04T01:59:43.842590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28782232121,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2404387,"num_read_kb":8823189,"num_write":2243919,"num_write_kb":8809215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100660,"ondisk_log_size":100660,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22610288,"kb_used_data":17514580,"kb_used_omap":5327,"kb_used_meta":5090352,"kb_avail":260505232,"statfs":{"total":289910292480,"available":266757357568,"internally_reserved":0,"allocated":17934929920,"data_stored":37617408913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5455678,"internal_metadata":5212520642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,10,5,16,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20017152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4811,"num_read_kb":19192,"num_write":4832,"num_write_kb":19328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002025"},"pg_stats":[{"pgid":"2.7","version":"158'9076","reported_seq":12734,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.658784+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T01:59:39.658784+0000","last_peered":"2023-10-04T01:59:39.658784+0000","last_clean":"2023-10-04T01:59:39.658784+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T01:59:39.658784+0000","last_undegraded":"2023-10-04T01:59:39.658784+0000","last_fullsized":"2023-10-04T01:59:39.658784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'7795","reported_seq":11632,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.638617+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T01:59:39.638617+0000","last_peered":"2023-10-04T01:59:39.638617+0000","last_clean":"2023-10-04T01:59:39.638617+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T01:59:39.638617+0000","last_undegraded":"2023-10-04T01:59:39.638617+0000","last_fullsized":"2023-10-04T01:59:39.638617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'9225","reported_seq":13191,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.658611+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T01:59:39.658611+0000","last_peered":"2023-10-04T01:59:39.658611+0000","last_clean":"2023-10-04T01:59:39.658611+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T01:59:39.658611+0000","last_undegraded":"2023-10-04T01:59:39.658611+0000","last_fullsized":"2023-10-04T01:59:39.658611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'7655","reported_seq":12467,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.658651+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T01:59:39.658651+0000","last_peered":"2023-10-04T01:59:39.658651+0000","last_clean":"2023-10-04T01:59:39.658651+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T01:59:39.658651+0000","last_undegraded":"2023-10-04T01:59:39.658651+0000","last_fullsized":"2023-10-04T01:59:39.658651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7655,"log_dups_size":0,"ondisk_log_size":7655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'561903","reported_seq":1108481,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:43.185459+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T01:59:43.185459+0000","last_peered":"2023-10-04T01:59:43.185459+0000","last_clean":"2023-10-04T01:59:43.185459+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T01:59:43.185459+0000","last_undegraded":"2023-10-04T01:59:43.185459+0000","last_fullsized":"2023-10-04T01:59:43.185459+0000","mapping_epoch":15,"log_start":"157'551873","ondisk_log_start":"157'551873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7229284352,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":551474,"num_read_kb":2173880,"num_write":550354,"num_write_kb":2201416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'18301","reported_seq":23879,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:42.968521+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T01:59:42.968521+0000","last_peered":"2023-10-04T01:59:42.968521+0000","last_clean":"2023-10-04T01:59:42.968521+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T01:59:42.968521+0000","last_undegraded":"2023-10-04T01:59:42.968521+0000","last_fullsized":"2023-10-04T01:59:42.968521+0000","mapping_epoch":12,"log_start":"95'8300","ondisk_log_start":"95'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'549232","reported_seq":1083546,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:40.214274+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T01:59:40.214274+0000","last_peered":"2023-10-04T01:59:40.214274+0000","last_clean":"2023-10-04T01:59:40.214274+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T01:59:40.214274+0000","last_undegraded":"2023-10-04T01:59:40.214274+0000","last_fullsized":"2023-10-04T01:59:40.214274+0000","mapping_epoch":15,"log_start":"156'539167","ondisk_log_start":"156'539167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7076249600,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":539129,"num_read_kb":2125280,"num_write":537852,"num_write_kb":2151408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9584,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.671880+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T01:59:39.671880+0000","last_peered":"2023-10-04T01:59:39.671880+0000","last_clean":"2023-10-04T01:59:39.671880+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T01:59:39.671880+0000","last_undegraded":"2023-10-04T01:59:39.671880+0000","last_fullsized":"2023-10-04T01:59:39.671880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'573932","reported_seq":1132759,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:43.178939+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T01:59:43.178939+0000","last_peered":"2023-10-04T01:59:43.178939+0000","last_clean":"2023-10-04T01:59:43.178939+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T01:59:43.178939+0000","last_undegraded":"2023-10-04T01:59:43.178939+0000","last_fullsized":"2023-10-04T01:59:43.178939+0000","mapping_epoch":15,"log_start":"157'563866","ondisk_log_start":"157'563866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7403347968,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":563834,"num_read_kb":2223388,"num_write":562071,"num_write_kb":2248284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":9992,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.659196+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T01:59:39.659196+0000","last_peered":"2023-10-04T01:59:39.659196+0000","last_clean":"2023-10-04T01:59:39.659196+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T01:59:39.659196+0000","last_undegraded":"2023-10-04T01:59:39.659196+0000","last_fullsized":"2023-10-04T01:59:39.659196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'546866","reported_seq":1080854,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T01:59:40.212025+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T01:59:40.212025+0000","last_peered":"2023-10-04T01:59:40.212025+0000","last_clean":"2023-10-04T01:59:40.212025+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T01:59:40.212025+0000","last_undegraded":"2023-10-04T01:59:40.212025+0000","last_fullsized":"2023-10-04T01:59:40.212025+0000","mapping_epoch":15,"log_start":"157'536863","ondisk_log_start":"157'536863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":2905,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 404s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7072743443,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":538776,"num_read_kb":2123788,"num_write":537998,"num_write_kb":2151986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'19207","reported_seq":25012,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:39.658846+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T01:59:39.658846+0000","last_peered":"2023-10-04T01:59:39.658846+0000","last_clean":"2023-10-04T01:59:39.658846+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T01:59:39.658846+0000","last_undegraded":"2023-10-04T01:59:39.658846+0000","last_fullsized":"2023-10-04T01:59:39.658846+0000","mapping_epoch":12,"log_start":"95'9200","ondisk_log_start":"95'9200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":338,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-04T01:57:58.501390+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:57:58.501390+0000","last_peered":"2023-10-04T01:57:58.501390+0000","last_clean":"2023-10-04T01:57:58.501390+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:57:58.501390+0000","last_undegraded":"2023-10-04T01:57:58.501390+0000","last_fullsized":"2023-10-04T01:57:58.501390+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28781625363,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2193213,"num_read_kb":8646336,"num_write":2188275,"num_write_kb":8753094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17928777728,"data_stored":37615874086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40164,"ondisk_log_size":40164,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60397,"ondisk_log_size":60397,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739408,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10352312,"kb_used_data":8756332,"kb_used_omap":2777,"kb_used_meta":1593190,"kb_avail":84019528,"statfs":{"total":96636764160,"available":86035996672,"internally_reserved":0,"allocated":8966483968,"data_stored":18808710238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1631427165},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.47099999999999997,"15min":0.47899999999999998},"min":{"1min":0.192,"5min":0.158,"15min":0.158},"max":{"1min":4.665,"5min":4.665,"15min":6.7199999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.26200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Oct 4 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.44500000000000001,"15min":0.60199999999999998},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.437,"15min":0.55600000000000005},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":4.0659999999999998,"5min":4.0659999999999998,"15min":35.423999999999999},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739405,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6275384,"kb_used_data":4306096,"kb_used_omap":1191,"kb_used_meta":1968088,"kb_avail":88096456,"statfs":{"total":96636764160,"available":90210770944,"internally_reserved":0,"allocated":4409442304,"data_stored":9259166190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2015322489},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.309,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.65200000000000002,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.51200000000000001,"15min":0.54500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.8440000000000003,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.60299999999999998,"15min":0.51900000000000002},"min":{"1min":0.221,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9809999999999999,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.625,"15min":0.53700000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.378}]}]},{"osd":0,"up_from":8,"seq":34359739406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5982592,"kb_used_data":4452152,"kb_used_omap":1358,"kb_used_meta":1529073,"kb_avail":88389248,"statfs":{"total":96636764160,"available":90510589952,"internally_reserved":0,"allocated":4559003648,"data_stored":9549532485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1565770988},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41799999999999998,"15min":0.48299999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.51800000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46899999999999997,"15min":0.60099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.21099999999999999},"max":{"1min":0.69399999999999995,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.49199999999999999}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.46500000000000002,"15min":0.62},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":0.67200000000000004,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.6340000000000003,"5min":5.2309999999999999,"15min":14.051},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4556492800,"data_stored":9548468224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4406677504,"data_stored":9258250259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8965607424,"data_stored":18809155603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T01:59:45.826 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T01:59:45.826 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T01:59:46.520 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.510+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T01:59:46.521 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.514+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T01:59:46.535 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T01:59:46.536 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T01:59:46.536 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T01:59:46.526+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T02:00:05.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:00:06.327 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:00:06.327 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:00:06.379 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2763,"stamp":"2023-10-04T02:00:05.847252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28818547257,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2413077,"num_read_kb":8857813,"num_write":2252706,"num_write_kb":8844363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100876,"ondisk_log_size":100876,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22892856,"kb_used_data":17579808,"kb_used_omap":5327,"kb_used_meta":5307696,"kb_avail":260222664,"statfs":{"total":289910292480,"available":266468007936,"internally_reserved":0,"allocated":18001723392,"data_stored":37684202385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5455678,"internal_metadata":5435080898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,8,14,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20307968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4832,"num_read_kb":19244,"num_write":4900,"num_write_kb":19600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002045"},"pg_stats":[{"pgid":"2.7","version":"158'9098","reported_seq":12766,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:05.262945+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:00:05.262945+0000","last_peered":"2023-10-04T02:00:05.262945+0000","last_clean":"2023-10-04T02:00:05.262945+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:00:05.262945+0000","last_undegraded":"2023-10-04T02:00:05.262945+0000","last_fullsized":"2023-10-04T02:00:05.262945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'7806","reported_seq":11648,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:04.706708+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:00:04.706708+0000","last_peered":"2023-10-04T02:00:04.706708+0000","last_clean":"2023-10-04T02:00:04.706708+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:00:04.706708+0000","last_undegraded":"2023-10-04T02:00:04.706708+0000","last_fullsized":"2023-10-04T02:00:04.706708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7806,"log_dups_size":0,"ondisk_log_size":7806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'9247","reported_seq":13223,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:05.262839+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:00:05.262839+0000","last_peered":"2023-10-04T02:00:05.262839+0000","last_clean":"2023-10-04T02:00:05.262839+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:00:05.262839+0000","last_undegraded":"2023-10-04T02:00:05.262839+0000","last_fullsized":"2023-10-04T02:00:05.262839+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9247,"log_dups_size":0,"ondisk_log_size":9247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'7677","reported_seq":12499,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:05.262682+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:00:05.262682+0000","last_peered":"2023-10-04T02:00:05.262682+0000","last_clean":"2023-10-04T02:00:05.262682+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:00:05.262682+0000","last_undegraded":"2023-10-04T02:00:05.262682+0000","last_fullsized":"2023-10-04T02:00:05.262682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'563821","reported_seq":1112280,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:03.218658+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:00:03.218658+0000","last_peered":"2023-10-04T02:00:03.218658+0000","last_clean":"2023-10-04T02:00:03.218658+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:00:03.218658+0000","last_undegraded":"2023-10-04T02:00:03.218658+0000","last_fullsized":"2023-10-04T02:00:03.218658+0000","mapping_epoch":15,"log_start":"157'553773","ondisk_log_start":"157'553773","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7237197824,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":553355,"num_read_kb":2181384,"num_write":552272,"num_write_kb":2209088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"158'18334","reported_seq":23924,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:00.489581+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:00:00.489581+0000","last_peered":"2023-10-04T02:00:00.489581+0000","last_clean":"2023-10-04T02:00:00.489581+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:00:00.489581+0000","last_undegraded":"2023-10-04T02:00:00.489581+0000","last_fullsized":"2023-10-04T02:00:00.489581+0000","mapping_epoch":12,"log_start":"95'8300","ondisk_log_start":"95'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'551627","reported_seq":1088266,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:05.281267+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:00:05.281267+0000","last_peered":"2023-10-04T02:00:05.281267+0000","last_clean":"2023-10-04T02:00:05.281267+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:00:05.281267+0000","last_undegraded":"2023-10-04T02:00:05.281267+0000","last_fullsized":"2023-10-04T02:00:05.281267+0000","mapping_epoch":15,"log_start":"157'541567","ondisk_log_start":"157'541567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7086116864,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":541458,"num_read_kb":2134552,"num_write":540247,"num_write_kb":2160988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9600,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:00.145297+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:00:00.145297+0000","last_peered":"2023-10-04T02:00:00.145297+0000","last_clean":"2023-10-04T02:00:00.145297+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:00:00.145297+0000","last_undegraded":"2023-10-04T02:00:00.145297+0000","last_fullsized":"2023-10-04T02:00:00.145297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'575942","reported_seq":1136898,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:03.216378+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:00:03.216378+0000","last_peered":"2023-10-04T02:00:03.216378+0000","last_clean":"2023-10-04T02:00:03.216378+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:00:03.216378+0000","last_undegraded":"2023-10-04T02:00:03.216378+0000","last_fullsized":"2023-10-04T02:00:03.216378+0000","mapping_epoch":15,"log_start":"157'565866","ondisk_log_start":"157'565866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7411658752,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":565962,"num_read_kb":2231872,"num_write":564081,"num_write_kb":2256324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10005,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:00.143313+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:00:00.143313+0000","last_peered":"2023-10-04T02:00:00.143313+0000","last_clean":"2023-10-04T02:00:00.143313+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:00:00.143313+0000","last_undegraded":"2023-10-04T02:00:00.143313+0000","last_fullsized":"2023-10-04T02:00:00.143313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'549516","reported_seq":1085868,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:00:05.285715+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:00:05.285715+0000","last_peered":"2023-10-04T02:00:05.285715+0000","last_clean":"2023-10-04T02:00:05.285715+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:00:05.285715+0000","last_undegraded":"2023-10-04T02:00:05.285715+0000","last_fullsized":"2023-10-04T02:00:05.285715+0000","mapping_epoch":15,"log_start":"157'539463","ondisk_log_start":"157'539463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3137,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 429s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7082967059,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":541128,"num_read_kb":2133152,"num_write":540462,"num_write_kb":2161842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'19240","reported_seq":25060,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T02:00:05.263226+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:00:05.263226+0000","last_peered":"2023-10-04T02:00:05.263226+0000","last_clean":"2023-10-04T02:00:05.263226+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:00:05.263226+0000","last_undegraded":"2023-10-04T02:00:05.263226+0000","last_fullsized":"2023-10-04T02:00:05.263226+0000","mapping_epoch":12,"log_start":"95'9200","ondisk_log_start":"95'9200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":339,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:03.528473+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:59:03.528473+0000","last_peered":"2023-10-04T01:59:03.528473+0000","last_clean":"2023-10-04T01:59:03.528473+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:59:03.528473+0000","last_undegraded":"2023-10-04T01:59:03.528473+0000","last_fullsized":"2023-10-04T01:59:03.528473+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28817940499,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2201903,"num_read_kb":8680960,"num_write":2197062,"num_write_kb":8788242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17996783616,"data_stored":37683879974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40237,"ondisk_log_size":40237,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60540,"ondisk_log_size":60540,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739412,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10479252,"kb_used_data":8787148,"kb_used_omap":2777,"kb_used_meta":1689318,"kb_avail":83892588,"statfs":{"total":96636764160,"available":85906010112,"internally_reserved":0,"allocated":8998039552,"data_stored":18840265822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1729862237},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.47099999999999997,"15min":0.47899999999999998},"min":{"1min":0.192,"5min":0.158,"15min":0.158},"max":{"1min":4.665,"5min":4.665,"15min":6.7199999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.26200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.7569999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.42799999999999999,"15min":0.59999999999999998},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.46300000000000002,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.308,"5min":0.443,"15min":0.54800000000000004},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":0.45600000000000002,"5min":4.0659999999999998,"15min":35.423999999999999},"last":0.85599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739410,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6359564,"kb_used_data":4325380,"kb_used_omap":1191,"kb_used_meta":2032984,"kb_avail":88012276,"statfs":{"total":96636764160,"available":90124570624,"internally_reserved":0,"allocated":4429189120,"data_stored":9278913006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2081775993},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.309,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.65200000000000002,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.248},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.51200000000000001,"15min":0.54500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.8440000000000003,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.60299999999999998,"15min":0.51900000000000002},"min":{"1min":0.221,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9809999999999999,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.625,"15min":0.53700000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.36599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6054040,"kb_used_data":4467280,"kb_used_omap":1358,"kb_used_meta":1585393,"kb_avail":88317800,"statfs":{"total":96636764160,"available":90437427200,"internally_reserved":0,"allocated":4574494720,"data_stored":9565023557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1623442668},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.40100000000000002,"15min":0.47799999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.44900000000000001,"5min":5.4850000000000003,"15min":17.991},"last":0.316},{"interface":"front","average":{"1min":0.374,"5min":0.46999999999999997,"15min":0.59599999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.63700000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.47599999999999998}]},{"osd":2,"last update":"Wed Oct 4 01:59:07 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.46500000000000002,"15min":0.62},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22700000000000001},"max":{"1min":0.67200000000000004,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.6340000000000003,"5min":5.2309999999999999,"15min":14.051},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4572581888,"data_stored":9564557312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4426588160,"data_stored":9278160915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8997613568,"data_stored":18841161747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:00:06.381 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:00:06.381 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:00:08.063 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.050+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:00:08.067 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:00:08.067 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:00:08.068 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T02:00:08.068 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:00:08.068 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T02:00:08.068 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.058+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:00:08.072 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.066+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:00:08.072 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:08.066+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T02:00:26.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:00:26.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:00:26.917 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:00:26.946 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2774,"stamp":"2023-10-04T02:00:25.850614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28850778681,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2420970,"num_read_kb":8889297,"num_write":2260459,"num_write_kb":8875375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101066,"ondisk_log_size":101066,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23184924,"kb_used_data":17642564,"kb_used_omap":5327,"kb_used_meta":5537008,"kb_avail":259930596,"statfs":{"total":289910292480,"available":266168930304,"internally_reserved":0,"allocated":18065985536,"data_stored":37748451842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5455678,"internal_metadata":5669896386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,8,10,25,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20434944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4972,"num_read_kb":19832,"num_write":4898,"num_write_kb":19592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001992"},"pg_stats":[{"pgid":"2.7","version":"159'9132","reported_seq":12807,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:25.137524+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:00:25.137524+0000","last_peered":"2023-10-04T02:00:25.137524+0000","last_clean":"2023-10-04T02:00:25.137524+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:00:25.137524+0000","last_undegraded":"2023-10-04T02:00:25.137524+0000","last_fullsized":"2023-10-04T02:00:25.137524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'7823","reported_seq":11670,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:24.714341+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:00:24.714341+0000","last_peered":"2023-10-04T02:00:24.714341+0000","last_clean":"2023-10-04T02:00:24.714341+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:00:24.714341+0000","last_undegraded":"2023-10-04T02:00:24.714341+0000","last_fullsized":"2023-10-04T02:00:24.714341+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7823,"log_dups_size":0,"ondisk_log_size":7823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'9281","reported_seq":13264,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:25.136764+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:00:25.136764+0000","last_peered":"2023-10-04T02:00:25.136764+0000","last_clean":"2023-10-04T02:00:25.136764+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:00:25.136764+0000","last_undegraded":"2023-10-04T02:00:25.136764+0000","last_fullsized":"2023-10-04T02:00:25.136764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'7711","reported_seq":12540,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:25.135099+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:00:25.135099+0000","last_peered":"2023-10-04T02:00:25.135099+0000","last_clean":"2023-10-04T02:00:25.135099+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:00:25.135099+0000","last_undegraded":"2023-10-04T02:00:25.135099+0000","last_fullsized":"2023-10-04T02:00:25.135099+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"159'565760","reported_seq":1116249,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:23.284381+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:00:23.284381+0000","last_peered":"2023-10-04T02:00:23.284381+0000","last_clean":"2023-10-04T02:00:23.284381+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:00:23.284381+0000","last_undegraded":"2023-10-04T02:00:23.284381+0000","last_fullsized":"2023-10-04T02:00:23.284381+0000","mapping_epoch":15,"log_start":"157'555673","ondisk_log_start":"157'555673","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7245254656,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":555384,"num_read_kb":2189484,"num_write":554211,"num_write_kb":2216844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'18358","reported_seq":23961,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:22.462759+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:00:22.462759+0000","last_peered":"2023-10-04T02:00:22.462759+0000","last_clean":"2023-10-04T02:00:22.462759+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:00:22.462759+0000","last_undegraded":"2023-10-04T02:00:22.462759+0000","last_fullsized":"2023-10-04T02:00:22.462759+0000","mapping_epoch":12,"log_start":"95'8300","ondisk_log_start":"95'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"159'553559","reported_seq":1092123,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:25.362970+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:00:25.362970+0000","last_peered":"2023-10-04T02:00:25.362970+0000","last_clean":"2023-10-04T02:00:25.362970+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:00:25.362970+0000","last_undegraded":"2023-10-04T02:00:25.362970+0000","last_fullsized":"2023-10-04T02:00:25.362970+0000","mapping_epoch":15,"log_start":"157'543467","ondisk_log_start":"157'543467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7094091776,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":543381,"num_read_kb":2142232,"num_write":542179,"num_write_kb":2168716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9617,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:20.618996+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:00:20.618996+0000","last_peered":"2023-10-04T02:00:20.618996+0000","last_clean":"2023-10-04T02:00:20.618996+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:00:20.618996+0000","last_undegraded":"2023-10-04T02:00:20.618996+0000","last_fullsized":"2023-10-04T02:00:20.618996+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"159'577916","reported_seq":1140938,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:23.279762+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:00:23.279762+0000","last_peered":"2023-10-04T02:00:23.279762+0000","last_clean":"2023-10-04T02:00:23.279762+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:00:23.279762+0000","last_undegraded":"2023-10-04T02:00:23.279762+0000","last_fullsized":"2023-10-04T02:00:23.279762+0000","mapping_epoch":15,"log_start":"157'567866","ondisk_log_start":"157'567866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7419822080,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":568027,"num_read_kb":2240112,"num_write":566055,"num_write_kb":2264220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10018,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:20.619543+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:00:20.619543+0000","last_peered":"2023-10-04T02:00:20.619543+0000","last_clean":"2023-10-04T02:00:20.619543+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:00:20.619543+0000","last_undegraded":"2023-10-04T02:00:20.619543+0000","last_fullsized":"2023-10-04T02:00:20.619543+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"159'551567","reported_seq":1089812,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:00:25.364214+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:00:25.364214+0000","last_peered":"2023-10-04T02:00:25.364214+0000","last_clean":"2023-10-04T02:00:25.364214+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:00:25.364214+0000","last_undegraded":"2023-10-04T02:00:25.364214+0000","last_fullsized":"2023-10-04T02:00:25.364214+0000","mapping_epoch":15,"log_start":"157'541563","ondisk_log_start":"157'541563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3313,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 449s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7091003411,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":543004,"num_read_kb":2140616,"num_write":542370,"num_write_kb":2169474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'19291","reported_seq":25121,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:25.158941+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:00:25.158941+0000","last_peered":"2023-10-04T02:00:25.158941+0000","last_clean":"2023-10-04T02:00:25.158941+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:00:25.158941+0000","last_undegraded":"2023-10-04T02:00:25.158941+0000","last_fullsized":"2023-10-04T02:00:25.158941+0000","mapping_epoch":12,"log_start":"95'9200","ondisk_log_start":"95'9200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":340,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:45.229779+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:59:45.229779+0000","last_peered":"2023-10-04T01:59:45.229779+0000","last_clean":"2023-10-04T01:59:45.229779+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:59:45.229779+0000","last_undegraded":"2023-10-04T01:59:45.229779+0000","last_fullsized":"2023-10-04T01:59:45.229779+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28850171923,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2209796,"num_read_kb":8712444,"num_write":2204815,"num_write_kb":8819254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18060513280,"data_stored":37747609638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40233,"ondisk_log_size":40233,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60734,"ondisk_log_size":60734,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739416,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10624636,"kb_used_data":8818292,"kb_used_omap":2777,"kb_used_meta":1803558,"kb_avail":83747204,"statfs":{"total":96636764160,"available":85757136896,"internally_reserved":0,"allocated":9029931008,"data_stored":18872153049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1846843997},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.42399999999999999,"15min":0.48599999999999999},"min":{"1min":0.22800000000000001,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.375,"5min":4.665,"15min":6.7199999999999998},"last":0.183},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47899999999999998,"15min":0.54300000000000004},"min":{"1min":0.29399999999999998,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.4119999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.42799999999999999,"15min":0.59999999999999998},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.46300000000000002,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.443,"15min":0.54800000000000004},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":0.45600000000000002,"5min":4.0659999999999998,"15min":35.423999999999999},"last":0.24099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739414,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6435920,"kb_used_data":4340808,"kb_used_omap":1191,"kb_used_meta":2093912,"kb_avail":87935920,"statfs":{"total":96636764160,"available":90046382080,"internally_reserved":0,"allocated":4444987392,"data_stored":9294707049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1220231,"internal_metadata":2144166265},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,18,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.308,"15min":0.32000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.78000000000000003,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.54100000000000004},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.242,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.26600000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57599999999999996,"15min":0.52800000000000002},"min":{"1min":0.20899999999999999,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9460000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.373},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":3.125,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6124368,"kb_used_data":4483464,"kb_used_omap":1358,"kb_used_meta":1639537,"kb_avail":88247472,"statfs":{"total":96636764160,"available":90365411328,"internally_reserved":0,"allocated":4591067136,"data_stored":9581591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1678886124},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.40100000000000002,"15min":0.47799999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.44900000000000001,"5min":5.4850000000000003,"15min":17.991},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.46999999999999997,"15min":0.59599999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.63700000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47899999999999998,"15min":0.60699999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.3650000000000002,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.55200000000000005,"5min":5.2309999999999999,"15min":14.051},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4588609536,"data_stored":9580584960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4442320896,"data_stored":9293893651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9029582848,"data_stored":18873131027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:00:26.948 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:00:26.949 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:00:46.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:00:47.413 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:00:47.414 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:00:47.437 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2784,"stamp":"2023-10-04T02:00:45.857036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28883964473,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2428765,"num_read_kb":8920405,"num_write":2268475,"num_write_kb":8907439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101008,"ondisk_log_size":101008,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23127704,"kb_used_data":17706560,"kb_used_omap":5329,"kb_used_meta":5415790,"kb_avail":259987816,"statfs":{"total":289910292480,"available":266227523584,"internally_reserved":0,"allocated":18131517440,"data_stored":37813983746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5457357,"internal_metadata":5545769523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,8,4,4,12,18],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20434944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5008,"num_read_kb":19992,"num_write":4948,"num_write_kb":19792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005036"},"pg_stats":[{"pgid":"2.7","version":"159'9150","reported_seq":12834,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.819742+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:00:43.819742+0000","last_peered":"2023-10-04T02:00:43.819742+0000","last_clean":"2023-10-04T02:00:43.819742+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:00:43.819742+0000","last_undegraded":"2023-10-04T02:00:43.819742+0000","last_fullsized":"2023-10-04T02:00:43.819742+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9150,"log_dups_size":0,"ondisk_log_size":9150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'7832","reported_seq":11683,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:41.099066+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:00:41.099066+0000","last_peered":"2023-10-04T02:00:41.099066+0000","last_clean":"2023-10-04T02:00:41.099066+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:00:41.099066+0000","last_undegraded":"2023-10-04T02:00:41.099066+0000","last_fullsized":"2023-10-04T02:00:41.099066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'9299","reported_seq":13290,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.819686+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:00:43.819686+0000","last_peered":"2023-10-04T02:00:43.819686+0000","last_clean":"2023-10-04T02:00:43.819686+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:00:43.819686+0000","last_undegraded":"2023-10-04T02:00:43.819686+0000","last_fullsized":"2023-10-04T02:00:43.819686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9299,"log_dups_size":0,"ondisk_log_size":9299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'7729","reported_seq":12566,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.814511+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:00:43.814511+0000","last_peered":"2023-10-04T02:00:43.814511+0000","last_clean":"2023-10-04T02:00:43.814511+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:00:43.814511+0000","last_undegraded":"2023-10-04T02:00:43.814511+0000","last_fullsized":"2023-10-04T02:00:43.814511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"159'567778","reported_seq":1120162,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.362120+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:00:43.362120+0000","last_peered":"2023-10-04T02:00:43.362120+0000","last_clean":"2023-10-04T02:00:43.362120+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:00:43.362120+0000","last_undegraded":"2023-10-04T02:00:43.362120+0000","last_fullsized":"2023-10-04T02:00:43.362120+0000","mapping_epoch":15,"log_start":"157'557773","ondisk_log_start":"157'557773","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7253524480,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":557277,"num_read_kb":2197024,"num_write":556229,"num_write_kb":2224916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'18388","reported_seq":24003,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:41.098855+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:00:41.098855+0000","last_peered":"2023-10-04T02:00:41.098855+0000","last_clean":"2023-10-04T02:00:41.098855+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:00:41.098855+0000","last_undegraded":"2023-10-04T02:00:41.098855+0000","last_fullsized":"2023-10-04T02:00:41.098855+0000","mapping_epoch":12,"log_start":"95'8300","ondisk_log_start":"95'8300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"159'555535","reported_seq":1096060,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:45.460311+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:00:45.460311+0000","last_peered":"2023-10-04T02:00:45.460311+0000","last_clean":"2023-10-04T02:00:45.460311+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:00:45.460311+0000","last_undegraded":"2023-10-04T02:00:45.460311+0000","last_fullsized":"2023-10-04T02:00:45.460311+0000","mapping_epoch":15,"log_start":"157'545467","ondisk_log_start":"157'545467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7102271488,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":545340,"num_read_kb":2150040,"num_write":544155,"num_write_kb":2176620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9633,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:41.108607+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:00:41.108607+0000","last_peered":"2023-10-04T02:00:41.108607+0000","last_clean":"2023-10-04T02:00:41.108607+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:00:41.108607+0000","last_undegraded":"2023-10-04T02:00:41.108607+0000","last_fullsized":"2023-10-04T02:00:41.108607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"159'579946","reported_seq":1144987,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.360653+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:00:43.360653+0000","last_peered":"2023-10-04T02:00:43.360653+0000","last_clean":"2023-10-04T02:00:43.360653+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:00:43.360653+0000","last_undegraded":"2023-10-04T02:00:43.360653+0000","last_fullsized":"2023-10-04T02:00:43.360653+0000","mapping_epoch":15,"log_start":"157'569866","ondisk_log_start":"157'569866","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7428333568,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":570047,"num_read_kb":2248192,"num_write":568085,"num_write_kb":2272340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10030,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:41.099423+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:00:41.099423+0000","last_peered":"2023-10-04T02:00:41.099423+0000","last_clean":"2023-10-04T02:00:41.099423+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:00:41.099423+0000","last_undegraded":"2023-10-04T02:00:41.099423+0000","last_fullsized":"2023-10-04T02:00:41.099423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"159'553665","reported_seq":1093844,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:00:45.461553+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:00:45.461553+0000","last_peered":"2023-10-04T02:00:45.461553+0000","last_clean":"2023-10-04T02:00:45.461553+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:00:45.461553+0000","last_undegraded":"2023-10-04T02:00:45.461553+0000","last_fullsized":"2023-10-04T02:00:45.461553+0000","mapping_epoch":15,"log_start":"158'543663","ondisk_log_start":"158'543663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3446,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 470s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7099228179,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":544927,"num_read_kb":2148296,"num_write":544362,"num_write_kb":2177442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'19318","reported_seq":25160,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:43.829844+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:00:43.829844+0000","last_peered":"2023-10-04T02:00:43.829844+0000","last_clean":"2023-10-04T02:00:43.829844+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:00:43.829844+0000","last_undegraded":"2023-10-04T02:00:43.829844+0000","last_fullsized":"2023-10-04T02:00:43.829844+0000","mapping_epoch":12,"log_start":"95'9300","ondisk_log_start":"95'9300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":340,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:45.229779+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:59:45.229779+0000","last_peered":"2023-10-04T01:59:45.229779+0000","last_clean":"2023-10-04T01:59:45.229779+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:59:45.229779+0000","last_undegraded":"2023-10-04T01:59:45.229779+0000","last_fullsized":"2023-10-04T01:59:45.229779+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28883357715,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2217591,"num_read_kb":8743552,"num_write":2212831,"num_write_kb":8851318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18126077952,"data_stored":37813174310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40155,"ondisk_log_size":40155,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60754,"ondisk_log_size":60754,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739420,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10741268,"kb_used_data":8850252,"kb_used_omap":2777,"kb_used_meta":1888230,"kb_avail":83630572,"statfs":{"total":96636764160,"available":85637705728,"internally_reserved":0,"allocated":9062658048,"data_stored":18904880089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":1933548125},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.42399999999999999,"15min":0.48599999999999999},"min":{"1min":0.22800000000000001,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.375,"5min":4.665,"15min":6.7199999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47899999999999998,"15min":0.54300000000000004},"min":{"1min":0.29399999999999998,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.4119999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Oct 4 01:59:53 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.42799999999999999,"15min":0.59999999999999998},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.46300000000000002,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.308,"5min":0.443,"15min":0.54800000000000004},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":0.45600000000000002,"5min":4.0659999999999998,"15min":35.423999999999999},"last":6.4249999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739418,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6205488,"kb_used_data":4356520,"kb_used_omap":1193,"kb_used_meta":1847766,"kb_avail":88166352,"statfs":{"total":96636764160,"available":90282344448,"internally_reserved":0,"allocated":4461076480,"data_stored":9310796137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":1892113130},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,4,12,18],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.308,"15min":0.32000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.78000000000000003,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.54100000000000004},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.242,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57599999999999996,"15min":0.52800000000000002},"min":{"1min":0.20899999999999999,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9460000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.192},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":3.125,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6180948,"kb_used_data":4499788,"kb_used_omap":1358,"kb_used_meta":1679793,"kb_avail":88190892,"statfs":{"total":96636764160,"available":90307473408,"internally_reserved":0,"allocated":4607782912,"data_stored":9598307520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1720108268},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.40100000000000002,"15min":0.47799999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.44900000000000001,"5min":5.4850000000000003,"15min":17.991},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.46999999999999997,"15min":0.59599999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.63700000000000001,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47899999999999998,"15min":0.60699999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.3650000000000002,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.37},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.55200000000000005,"5min":5.2309999999999999,"15min":14.051},"last":0.26600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4605190144,"data_stored":9597165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4458561536,"data_stored":9310134291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9062326272,"data_stored":18905874451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:00:47.439 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:00:47.439 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:00:50.513 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:00:50.514 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:00:50.514 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:00:50.514 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T02:00:50.514 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:00:50.515 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T02:00:50.515 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T02:00:50.515 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:00:50.515 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:00:50.515 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:00:50.505+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T02:01:04.950 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:01:04.944+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:01:07.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:01:07.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:01:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:01:07.985 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2795,"stamp":"2023-10-04T02:01:07.860802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28916499001,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2436619,"num_read_kb":8951721,"num_write":2276326,"num_write_kb":8938843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101037,"ondisk_log_size":101037,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23365960,"kb_used_data":17768560,"kb_used_omap":5329,"kb_used_meta":5592046,"kb_avail":259749560,"statfs":{"total":289910292480,"available":265983549440,"internally_reserved":0,"allocated":18195005440,"data_stored":37877471746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5457357,"internal_metadata":5726255667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,6,4,5,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16322560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3961,"num_read_kb":15800,"num_write":3963,"num_write_kb":15852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002017"},"pg_stats":[{"pgid":"2.7","version":"159'9172","reported_seq":12864,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.616489+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:01:01.616489+0000","last_peered":"2023-10-04T02:01:01.616489+0000","last_clean":"2023-10-04T02:01:01.616489+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:01:01.616489+0000","last_undegraded":"2023-10-04T02:01:01.616489+0000","last_fullsized":"2023-10-04T02:01:01.616489+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'7843","reported_seq":11698,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.578768+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:01:01.578768+0000","last_peered":"2023-10-04T02:01:01.578768+0000","last_clean":"2023-10-04T02:01:01.578768+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:01:01.578768+0000","last_undegraded":"2023-10-04T02:01:01.578768+0000","last_fullsized":"2023-10-04T02:01:01.578768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'9321","reported_seq":13320,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.616352+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:01:01.616352+0000","last_peered":"2023-10-04T02:01:01.616352+0000","last_clean":"2023-10-04T02:01:01.616352+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:01:01.616352+0000","last_undegraded":"2023-10-04T02:01:01.616352+0000","last_fullsized":"2023-10-04T02:01:01.616352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'7751","reported_seq":12596,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.606532+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:01:01.606532+0000","last_peered":"2023-10-04T02:01:01.606532+0000","last_clean":"2023-10-04T02:01:01.606532+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:01:01.606532+0000","last_undegraded":"2023-10-04T02:01:01.606532+0000","last_fullsized":"2023-10-04T02:01:01.606532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"159'569718","reported_seq":1124106,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:03.477997+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:01:03.477997+0000","last_peered":"2023-10-04T02:01:03.477997+0000","last_clean":"2023-10-04T02:01:03.477997+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:01:03.477997+0000","last_undegraded":"2023-10-04T02:01:03.477997+0000","last_fullsized":"2023-10-04T02:01:03.477997+0000","mapping_epoch":15,"log_start":"158'559673","ondisk_log_start":"158'559673","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7261659136,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":559282,"num_read_kb":2205032,"num_write":558169,"num_write_kb":2232676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'18418","reported_seq":24045,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:02.459390+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:01:02.459390+0000","last_peered":"2023-10-04T02:01:02.459390+0000","last_clean":"2023-10-04T02:01:02.459390+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:01:02.459390+0000","last_undegraded":"2023-10-04T02:01:02.459390+0000","last_fullsized":"2023-10-04T02:01:02.459390+0000","mapping_epoch":12,"log_start":"95'8400","ondisk_log_start":"95'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"159'557435","reported_seq":1099853,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:05.562979+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:01:05.562979+0000","last_peered":"2023-10-04T02:01:05.562979+0000","last_clean":"2023-10-04T02:01:05.562979+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:01:05.562979+0000","last_undegraded":"2023-10-04T02:01:05.562979+0000","last_fullsized":"2023-10-04T02:01:05.562979+0000","mapping_epoch":15,"log_start":"158'547367","ondisk_log_start":"158'547367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7110131712,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":547235,"num_read_kb":2157584,"num_write":546055,"num_write_kb":2184220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9650,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.579992+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:01:01.579992+0000","last_peered":"2023-10-04T02:01:01.579992+0000","last_clean":"2023-10-04T02:01:01.579992+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:01:01.579992+0000","last_undegraded":"2023-10-04T02:01:01.579992+0000","last_fullsized":"2023-10-04T02:01:01.579992+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"159'581970","reported_seq":1149015,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:03.470815+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:01:03.470815+0000","last_peered":"2023-10-04T02:01:03.470815+0000","last_clean":"2023-10-04T02:01:03.470815+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:01:03.470815+0000","last_undegraded":"2023-10-04T02:01:03.470815+0000","last_fullsized":"2023-10-04T02:01:03.470815+0000","mapping_epoch":15,"log_start":"158'571966","ondisk_log_start":"158'571966","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7436689408,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":572050,"num_read_kb":2256168,"num_write":570109,"num_write_kb":2280436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10042,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.579249+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:01:01.579249+0000","last_peered":"2023-10-04T02:01:01.579249+0000","last_clean":"2023-10-04T02:01:01.579249+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:01:01.579249+0000","last_undegraded":"2023-10-04T02:01:01.579249+0000","last_fullsized":"2023-10-04T02:01:01.579249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"159'555790","reported_seq":1097925,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:01:05.564830+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:01:05.564830+0000","last_peered":"2023-10-04T02:01:05.564830+0000","last_clean":"2023-10-04T02:01:05.564830+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:01:05.564830+0000","last_undegraded":"2023-10-04T02:01:05.564830+0000","last_fullsized":"2023-10-04T02:01:05.564830+0000","mapping_epoch":15,"log_start":"158'545763","ondisk_log_start":"158'545763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3617,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 490s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7107411987,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":546878,"num_read_kb":2156084,"num_write":546349,"num_write_kb":2185390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'19351","reported_seq":25205,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:01.627009+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:01:01.627009+0000","last_peered":"2023-10-04T02:01:01.627009+0000","last_clean":"2023-10-04T02:01:01.627009+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:01:01.627009+0000","last_undegraded":"2023-10-04T02:01:01.627009+0000","last_fullsized":"2023-10-04T02:01:01.627009+0000","mapping_epoch":12,"log_start":"95'9300","ondisk_log_start":"95'9300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":340,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-04T01:59:45.229779+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T01:59:45.229779+0000","last_peered":"2023-10-04T01:59:45.229779+0000","last_clean":"2023-10-04T01:59:45.229779+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T01:59:45.229779+0000","last_undegraded":"2023-10-04T01:59:45.229779+0000","last_fullsized":"2023-10-04T01:59:45.229779+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28915892243,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2225445,"num_read_kb":8774868,"num_write":2220682,"num_write_kb":8882722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18190221312,"data_stored":37877317670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40144,"ondisk_log_size":40144,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60794,"ondisk_log_size":60794,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10880872,"kb_used_data":8881632,"kb_used_omap":2777,"kb_used_meta":1996454,"kb_avail":83490968,"statfs":{"total":96636764160,"available":85494751232,"internally_reserved":0,"allocated":9094791168,"data_stored":18937013209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2844067,"internal_metadata":2044369501},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.42399999999999999,"15min":0.48599999999999999},"min":{"1min":0.22800000000000001,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.375,"5min":4.665,"15min":6.7199999999999998},"last":3.069},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47899999999999998,"15min":0.54300000000000004},"min":{"1min":0.29399999999999998,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.4119999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":3.1539999999999999}]},{"osd":1,"last update":"Wed Oct 4 02:00:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.42899999999999999,"15min":0.59799999999999998},"min":{"1min":0.24399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.625,"5min":4.7210000000000001,"15min":33.975000000000001},"last":3.1219999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.47799999999999998,"15min":0.56999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":35.423999999999999},"last":3.1859999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739422,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6257024,"kb_used_data":4371960,"kb_used_omap":1193,"kb_used_meta":1883862,"kb_avail":88114816,"statfs":{"total":96636764160,"available":90229571584,"internally_reserved":0,"allocated":4476887040,"data_stored":9326606697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":1929075434},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,3,3,5,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.308,"15min":0.32000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.78000000000000003,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.192},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.54100000000000004},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.242,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.57599999999999996,"15min":0.52800000000000002},"min":{"1min":0.20899999999999999,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":2.9460000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.313},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":3.125,"5min":4.7519999999999998,"15min":6.1050000000000004},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6228064,"kb_used_data":4514968,"kb_used_omap":1358,"kb_used_meta":1711729,"kb_avail":88143776,"statfs":{"total":96636764160,"available":90259226624,"internally_reserved":0,"allocated":4623327232,"data_stored":9613851840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1752810732},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.20300000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.8010000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.57799999999999996,"15min":0.629},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":4.851,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.254}]},{"osd":2,"last update":"Wed Oct 4 02:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47899999999999998,"15min":0.60699999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.3650000000000002,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.55200000000000005,"5min":5.2309999999999999,"15min":14.051},"last":0.34699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4621426688,"data_stored":9613402112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4474482688,"data_stored":9326055443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9094311936,"data_stored":18937860115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:01:07.987 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:01:07.988 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:01:27.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:01:28.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:01:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:01:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2806,"stamp":"2023-10-04T02:01:27.864242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28949426745,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2444400,"num_read_kb":8982765,"num_write":2284298,"num_write_kb":8970731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101304,"ondisk_log_size":101304,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23531316,"kb_used_data":17833368,"kb_used_omap":5331,"kb_used_meta":5692588,"kb_avail":259584204,"statfs":{"total":289910292480,"available":265814224896,"internally_reserved":0,"allocated":18261368832,"data_stored":37943830866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5459037,"internal_metadata":5829211043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,4,11,6,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16580608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3918,"num_read_kb":15632,"num_write":4032,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001704"},"pg_stats":[{"pgid":"2.7","version":"160'9204","reported_seq":12905,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:24.438508+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:01:24.438508+0000","last_peered":"2023-10-04T02:01:24.438508+0000","last_clean":"2023-10-04T02:01:24.438508+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:01:24.438508+0000","last_undegraded":"2023-10-04T02:01:24.438508+0000","last_fullsized":"2023-10-04T02:01:24.438508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9204,"log_dups_size":0,"ondisk_log_size":9204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'7859","reported_seq":11719,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.821492+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:01:23.821492+0000","last_peered":"2023-10-04T02:01:23.821492+0000","last_clean":"2023-10-04T02:01:23.821492+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:01:23.821492+0000","last_undegraded":"2023-10-04T02:01:23.821492+0000","last_fullsized":"2023-10-04T02:01:23.821492+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'9353","reported_seq":13362,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:24.438508+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:01:24.438508+0000","last_peered":"2023-10-04T02:01:24.438508+0000","last_clean":"2023-10-04T02:01:24.438508+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:01:24.438508+0000","last_undegraded":"2023-10-04T02:01:24.438508+0000","last_fullsized":"2023-10-04T02:01:24.438508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'7783","reported_seq":12638,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:24.435758+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:01:24.435758+0000","last_peered":"2023-10-04T02:01:24.435758+0000","last_clean":"2023-10-04T02:01:24.435758+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:01:24.435758+0000","last_undegraded":"2023-10-04T02:01:24.435758+0000","last_fullsized":"2023-10-04T02:01:24.435758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'571779","reported_seq":1128127,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.569196+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:01:23.569196+0000","last_peered":"2023-10-04T02:01:23.569196+0000","last_clean":"2023-10-04T02:01:23.569196+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:01:23.569196+0000","last_undegraded":"2023-10-04T02:01:23.569196+0000","last_fullsized":"2023-10-04T02:01:23.569196+0000","mapping_epoch":15,"log_start":"158'561773","ondisk_log_start":"158'561773","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7270109184,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":561245,"num_read_kb":2212852,"num_write":560230,"num_write_kb":2240920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'18447","reported_seq":24088,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.548291+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:01:23.548291+0000","last_peered":"2023-10-04T02:01:23.548291+0000","last_clean":"2023-10-04T02:01:23.548291+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:01:23.548291+0000","last_undegraded":"2023-10-04T02:01:23.548291+0000","last_fullsized":"2023-10-04T02:01:23.548291+0000","mapping_epoch":12,"log_start":"95'8400","ondisk_log_start":"95'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'559355","reported_seq":1103741,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:25.636377+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:01:25.636377+0000","last_peered":"2023-10-04T02:01:25.636377+0000","last_clean":"2023-10-04T02:01:25.636377+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:01:25.636377+0000","last_undegraded":"2023-10-04T02:01:25.636377+0000","last_fullsized":"2023-10-04T02:01:25.636377+0000","mapping_epoch":15,"log_start":"158'549267","ondisk_log_start":"158'549267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7118077952,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":549198,"num_read_kb":2165416,"num_write":547975,"num_write_kb":2191900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9667,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:22.068354+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:01:22.068354+0000","last_peered":"2023-10-04T02:01:22.068354+0000","last_clean":"2023-10-04T02:01:22.068354+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:01:22.068354+0000","last_undegraded":"2023-10-04T02:01:22.068354+0000","last_fullsized":"2023-10-04T02:01:22.068354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'584021","reported_seq":1152984,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.559700+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:01:23.559700+0000","last_peered":"2023-10-04T02:01:23.559700+0000","last_clean":"2023-10-04T02:01:23.559700+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:01:23.559700+0000","last_undegraded":"2023-10-04T02:01:23.559700+0000","last_fullsized":"2023-10-04T02:01:23.559700+0000","mapping_epoch":15,"log_start":"158'573966","ondisk_log_start":"158'573966","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7445196800,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":573966,"num_read_kb":2263816,"num_write":572160,"num_write_kb":2288640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10055,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:01:22.059336+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:01:22.059336+0000","last_peered":"2023-10-04T02:01:22.059336+0000","last_clean":"2023-10-04T02:01:22.059336+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:01:22.059336+0000","last_undegraded":"2023-10-04T02:01:22.059336+0000","last_fullsized":"2023-10-04T02:01:22.059336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'557836","reported_seq":1101925,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:01:25.651413+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:01:25.651413+0000","last_peered":"2023-10-04T02:01:25.651413+0000","last_clean":"2023-10-04T02:01:25.651413+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:01:25.651413+0000","last_undegraded":"2023-10-04T02:01:25.651413+0000","last_fullsized":"2023-10-04T02:01:25.651413+0000","mapping_epoch":15,"log_start":"158'547763","ondisk_log_start":"158'547763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3749,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 510s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7115436051,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":548817,"num_read_kb":2163828,"num_write":548289,"num_write_kb":2193150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'19399","reported_seq":25267,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:24.446378+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:01:24.446378+0000","last_peered":"2023-10-04T02:01:24.446378+0000","last_clean":"2023-10-04T02:01:24.446378+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:01:24.446378+0000","last_undegraded":"2023-10-04T02:01:24.446378+0000","last_fullsized":"2023-10-04T02:01:24.446378+0000","mapping_epoch":12,"log_start":"95'9300","ondisk_log_start":"95'9300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":341,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-04T02:00:13.545175+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:00:13.545175+0000","last_peered":"2023-10-04T02:00:13.545175+0000","last_clean":"2023-10-04T02:00:13.545175+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:00:13.545175+0000","last_undegraded":"2023-10-04T02:00:13.545175+0000","last_fullsized":"2023-10-04T02:00:13.545175+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28948819987,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2233226,"num_read_kb":8805912,"num_write":2228654,"num_write_kb":8914610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18255708160,"data_stored":37942804518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40222,"ondisk_log_size":40222,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60983,"ondisk_log_size":60983,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739428,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10901168,"kb_used_data":8913832,"kb_used_omap":2779,"kb_used_meta":1984548,"kb_avail":83470672,"statfs":{"total":96636764160,"available":85473968128,"internally_reserved":0,"allocated":9127763968,"data_stored":18969986009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":2032178125},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.48699999999999999},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.069,"5min":4.665,"15min":6.7199999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.50800000000000001,"15min":0.54400000000000004},"min":{"1min":0.246,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.1539999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Oct 4 02:00:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.42899999999999999,"15min":0.59799999999999998},"min":{"1min":0.24399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.625,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.47799999999999998,"15min":0.56999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.38200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739426,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6308644,"kb_used_data":4387480,"kb_used_omap":1193,"kb_used_meta":1919958,"kb_avail":88063196,"statfs":{"total":96636764160,"available":90176712704,"internally_reserved":0,"allocated":4492779520,"data_stored":9342494905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":1966037738},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,0,1,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.307,"15min":0.313},"min":{"1min":0.192,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.54600000000000004,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":2.0459999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.47199999999999998,"15min":0.53300000000000003},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.57599999999999996,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":2.0880000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":0.52400000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6321504,"kb_used_data":4532056,"kb_used_omap":1358,"kb_used_meta":1788081,"kb_avail":88050336,"statfs":{"total":96636764160,"available":90163544064,"internally_reserved":0,"allocated":4640825344,"data_stored":9631349952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1830995180},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,6,6,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.20300000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.8010000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.503},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.57799999999999996,"15min":0.629},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":4.851,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Oct 4 02:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.867,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51500000000000001,"15min":0.439},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4489999999999998,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.73499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4638269440,"data_stored":9630244864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4490317824,"data_stored":9341890579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9127120896,"data_stored":18970669075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:01:28.524 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:01:28.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T02:01:28.995 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T02:01:29.025 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:01:49.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:01:49.465 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:01:49.465 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:01:49.494 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2816,"stamp":"2023-10-04T02:01:47.867714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28982313529,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2452247,"num_read_kb":9014053,"num_write":2292268,"num_write_kb":9002611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101241,"ondisk_log_size":101241,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23293828,"kb_used_data":17896940,"kb_used_omap":5331,"kb_used_meta":5391532,"kb_avail":259821692,"statfs":{"total":289910292480,"available":266057412608,"internally_reserved":0,"allocated":18326466560,"data_stored":38008920050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5459037,"internal_metadata":5520929699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,7,9,5,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16547840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3871,"num_read_kb":15424,"num_write":3996,"num_write_kb":15984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"2.7","version":"160'9210","reported_seq":12919,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.539674+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:01:42.539674+0000","last_peered":"2023-10-04T02:01:42.539674+0000","last_clean":"2023-10-04T02:01:42.539674+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:01:42.539674+0000","last_undegraded":"2023-10-04T02:01:42.539674+0000","last_fullsized":"2023-10-04T02:01:42.539674+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9210,"log_dups_size":0,"ondisk_log_size":9210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'7862","reported_seq":11726,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:41.554725+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:01:41.554725+0000","last_peered":"2023-10-04T02:01:41.554725+0000","last_clean":"2023-10-04T02:01:41.554725+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:01:41.554725+0000","last_undegraded":"2023-10-04T02:01:41.554725+0000","last_fullsized":"2023-10-04T02:01:41.554725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'9359","reported_seq":13376,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.539288+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:01:42.539288+0000","last_peered":"2023-10-04T02:01:42.539288+0000","last_clean":"2023-10-04T02:01:42.539288+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:01:42.539288+0000","last_undegraded":"2023-10-04T02:01:42.539288+0000","last_fullsized":"2023-10-04T02:01:42.539288+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'7789","reported_seq":12652,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.538637+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:01:42.538637+0000","last_peered":"2023-10-04T02:01:42.538637+0000","last_clean":"2023-10-04T02:01:42.538637+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:01:42.538637+0000","last_undegraded":"2023-10-04T02:01:42.538637+0000","last_fullsized":"2023-10-04T02:01:42.538637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'573768","reported_seq":1132050,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:43.627807+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:01:43.627807+0000","last_peered":"2023-10-04T02:01:43.627807+0000","last_clean":"2023-10-04T02:01:43.627807+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:01:43.627807+0000","last_undegraded":"2023-10-04T02:01:43.627807+0000","last_fullsized":"2023-10-04T02:01:43.627807+0000","mapping_epoch":15,"log_start":"158'563673","ondisk_log_start":"158'563673","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7278313472,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":563173,"num_read_kb":2220548,"num_write":562219,"num_write_kb":2248876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'18499","reported_seq":24152,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.539149+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:01:42.539149+0000","last_peered":"2023-10-04T02:01:42.539149+0000","last_clean":"2023-10-04T02:01:42.539149+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:01:42.539149+0000","last_undegraded":"2023-10-04T02:01:42.539149+0000","last_fullsized":"2023-10-04T02:01:42.539149+0000","mapping_epoch":12,"log_start":"95'8400","ondisk_log_start":"95'8400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'561298","reported_seq":1107664,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:45.745605+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:01:45.745605+0000","last_peered":"2023-10-04T02:01:45.745605+0000","last_clean":"2023-10-04T02:01:45.745605+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:01:45.745605+0000","last_undegraded":"2023-10-04T02:01:45.745605+0000","last_fullsized":"2023-10-04T02:01:45.745605+0000","mapping_epoch":15,"log_start":"158'551267","ondisk_log_start":"158'551267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7126073344,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":551178,"num_read_kb":2173316,"num_write":549918,"num_write_kb":2199672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9683,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.569505+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:01:42.569505+0000","last_peered":"2023-10-04T02:01:42.569505+0000","last_clean":"2023-10-04T02:01:42.569505+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:01:42.569505+0000","last_undegraded":"2023-10-04T02:01:42.569505+0000","last_fullsized":"2023-10-04T02:01:42.569505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'586097","reported_seq":1157050,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:43.640096+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:01:43.640096+0000","last_peered":"2023-10-04T02:01:43.640096+0000","last_clean":"2023-10-04T02:01:43.640096+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:01:43.640096+0000","last_undegraded":"2023-10-04T02:01:43.640096+0000","last_fullsized":"2023-10-04T02:01:43.640096+0000","mapping_epoch":15,"log_start":"158'576066","ondisk_log_start":"158'576066","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7453802496,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":575956,"num_read_kb":2271752,"num_write":574236,"num_write_kb":2296944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10067,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.539584+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:01:42.539584+0000","last_peered":"2023-10-04T02:01:42.539584+0000","last_clean":"2023-10-04T02:01:42.539584+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:01:42.539584+0000","last_undegraded":"2023-10-04T02:01:42.539584+0000","last_fullsized":"2023-10-04T02:01:42.539584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'559883","reported_seq":1105937,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:01:45.753612+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:01:45.753612+0000","last_peered":"2023-10-04T02:01:45.753612+0000","last_clean":"2023-10-04T02:01:45.753612+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:01:45.753612+0000","last_undegraded":"2023-10-04T02:01:45.753612+0000","last_fullsized":"2023-10-04T02:01:45.753612+0000","mapping_epoch":15,"log_start":"158'549863","ondisk_log_start":"158'549863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3855,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 530s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7123517459,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":550766,"num_read_kb":2171584,"num_write":550251,"num_write_kb":2200998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'19408","reported_seq":25288,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:42.539808+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:01:42.539808+0000","last_peered":"2023-10-04T02:01:42.539808+0000","last_clean":"2023-10-04T02:01:42.539808+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:01:42.539808+0000","last_undegraded":"2023-10-04T02:01:42.539808+0000","last_fullsized":"2023-10-04T02:01:42.539808+0000","mapping_epoch":12,"log_start":"97'9400","ondisk_log_start":"97'9400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":342,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.568513+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:23.568513+0000","last_peered":"2023-10-04T02:01:23.568513+0000","last_clean":"2023-10-04T02:01:23.568513+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:23.568513+0000","last_undegraded":"2023-10-04T02:01:23.568513+0000","last_fullsized":"2023-10-04T02:01:23.568513+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28981706771,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2241073,"num_read_kb":8837200,"num_write":2236624,"num_write_kb":8946490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18320801792,"data_stored":38007898150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40177,"ondisk_log_size":40177,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":60965,"ondisk_log_size":60965,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739432,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10487416,"kb_used_data":8945520,"kb_used_omap":2779,"kb_used_meta":1539108,"kb_avail":83884424,"statfs":{"total":96636764160,"available":85897650176,"internally_reserved":0,"allocated":9160212480,"data_stored":19002430249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":1576047565},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.48699999999999999},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.069,"5min":4.665,"15min":6.7199999999999998},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.50800000000000001,"15min":0.54400000000000004},"min":{"1min":0.246,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.1539999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.28499999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:00:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.42899999999999999,"15min":0.59799999999999998},"min":{"1min":0.24399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.625,"5min":4.7210000000000001,"15min":33.975000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.47799999999999998,"15min":0.56999999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.49099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739431,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6378308,"kb_used_data":4403004,"kb_used_omap":1193,"kb_used_meta":1974102,"kb_avail":87993532,"statfs":{"total":96636764160,"available":90105376768,"internally_reserved":0,"allocated":4508676096,"data_stored":9358391481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":2021481194},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,5,2,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.307,"15min":0.313},"min":{"1min":0.192,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.54600000000000004,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.47199999999999998,"15min":0.53300000000000003},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.57599999999999996,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":0.52400000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.439},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6428104,"kb_used_data":4548416,"kb_used_omap":1358,"kb_used_meta":1878321,"kb_avail":87943736,"statfs":{"total":96636764160,"available":90054385664,"internally_reserved":0,"allocated":4657577984,"data_stored":9648098320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1923400940},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.20300000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.8010000000000002,"5min":5.4850000000000003,"15min":17.991},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.57799999999999996,"15min":0.629},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":4.851,"5min":5.5110000000000001,"15min":17.957000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.867,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51500000000000001,"15min":0.439},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4489999999999998,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.34899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4654936064,"data_stored":9646911488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4506288128,"data_stored":9357860883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9159577600,"data_stored":19003125779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:01:49.495 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:01:49.496 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:02:09.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:02:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:02:09.987 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:02:10.020 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2827,"stamp":"2023-10-04T02:02:09.871823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29018956345,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2461175,"num_read_kb":9049653,"num_write":2301160,"num_write_kb":9038179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101441,"ondisk_log_size":101441,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23597440,"kb_used_data":17972904,"kb_used_omap":5331,"kb_used_meta":5619180,"kb_avail":259518080,"statfs":{"total":289910292480,"available":265746513920,"internally_reserved":0,"allocated":18404253696,"data_stored":38086707186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5459037,"internal_metadata":5754041251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,8,9,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20381696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4930,"num_read_kb":19680,"num_write":4938,"num_write_kb":19752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002296"},"pg_stats":[{"pgid":"2.7","version":"160'9234","reported_seq":12951,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:04.709780+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:02:04.709780+0000","last_peered":"2023-10-04T02:02:04.709780+0000","last_clean":"2023-10-04T02:02:04.709780+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:02:04.709780+0000","last_undegraded":"2023-10-04T02:02:04.709780+0000","last_fullsized":"2023-10-04T02:02:04.709780+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'7874","reported_seq":11742,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:03.018944+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:02:03.018944+0000","last_peered":"2023-10-04T02:02:03.018944+0000","last_clean":"2023-10-04T02:02:03.018944+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:02:03.018944+0000","last_undegraded":"2023-10-04T02:02:03.018944+0000","last_fullsized":"2023-10-04T02:02:03.018944+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'9383","reported_seq":13408,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:04.716549+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:02:04.716549+0000","last_peered":"2023-10-04T02:02:04.716549+0000","last_clean":"2023-10-04T02:02:04.716549+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:02:04.716549+0000","last_undegraded":"2023-10-04T02:02:04.716549+0000","last_fullsized":"2023-10-04T02:02:04.716549+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9383,"log_dups_size":0,"ondisk_log_size":9383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'7813","reported_seq":12684,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:04.702674+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:02:04.702674+0000","last_peered":"2023-10-04T02:02:04.702674+0000","last_clean":"2023-10-04T02:02:04.702674+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:02:04.702674+0000","last_undegraded":"2023-10-04T02:02:04.702674+0000","last_fullsized":"2023-10-04T02:02:04.702674+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7813,"log_dups_size":0,"ondisk_log_size":7813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'576362","reported_seq":1137182,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:08.705156+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:02:08.705156+0000","last_peered":"2023-10-04T02:02:08.705156+0000","last_clean":"2023-10-04T02:02:08.705156+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:02:08.705156+0000","last_undegraded":"2023-10-04T02:02:08.705156+0000","last_fullsized":"2023-10-04T02:02:08.705156+0000","mapping_epoch":15,"log_start":"159'566273","ondisk_log_start":"159'566273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7288979456,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":565712,"num_read_kb":2230680,"num_write":564813,"num_write_kb":2259252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'18571","reported_seq":24239,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:08.572773+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:02:08.572773+0000","last_peered":"2023-10-04T02:02:08.572773+0000","last_clean":"2023-10-04T02:02:08.572773+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:02:08.572773+0000","last_undegraded":"2023-10-04T02:02:08.572773+0000","last_fullsized":"2023-10-04T02:02:08.572773+0000","mapping_epoch":12,"log_start":"95'8500","ondisk_log_start":"95'8500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'563163","reported_seq":1111447,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:05.805308+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:02:05.805308+0000","last_peered":"2023-10-04T02:02:05.805308+0000","last_clean":"2023-10-04T02:02:05.805308+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:02:05.805308+0000","last_undegraded":"2023-10-04T02:02:05.805308+0000","last_fullsized":"2023-10-04T02:02:05.805308+0000","mapping_epoch":15,"log_start":"159'553067","ondisk_log_start":"159'553067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7133802496,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":553096,"num_read_kb":2180956,"num_write":551783,"num_write_kb":2207132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9703,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:08.139262+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:02:08.139262+0000","last_peered":"2023-10-04T02:02:08.139262+0000","last_clean":"2023-10-04T02:02:08.139262+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:02:08.139262+0000","last_undegraded":"2023-10-04T02:02:08.139262+0000","last_fullsized":"2023-10-04T02:02:08.139262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'588621","reported_seq":1162106,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:08.702386+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:02:08.702386+0000","last_peered":"2023-10-04T02:02:08.702386+0000","last_clean":"2023-10-04T02:02:08.702386+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:02:08.702386+0000","last_undegraded":"2023-10-04T02:02:08.702386+0000","last_fullsized":"2023-10-04T02:02:08.702386+0000","mapping_epoch":15,"log_start":"159'578566","ondisk_log_start":"159'578566","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7464140800,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":578487,"num_read_kb":2281852,"num_write":576760,"num_write_kb":2307040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10083,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:08.139181+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:02:08.139181+0000","last_peered":"2023-10-04T02:02:08.139181+0000","last_clean":"2023-10-04T02:02:08.139181+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:02:08.139181+0000","last_undegraded":"2023-10-04T02:02:08.139181+0000","last_fullsized":"2023-10-04T02:02:08.139181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'561908","reported_seq":1109913,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:02:05.807078+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:02:05.807078+0000","last_peered":"2023-10-04T02:02:05.807078+0000","last_clean":"2023-10-04T02:02:05.807078+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:02:05.807078+0000","last_undegraded":"2023-10-04T02:02:05.807078+0000","last_fullsized":"2023-10-04T02:02:05.807078+0000","mapping_epoch":15,"log_start":"159'551863","ondisk_log_start":"159'551863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":3997,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 550s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7131426835,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":552706,"num_read_kb":2179312,"num_write":552160,"num_write_kb":2208634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'19444","reported_seq":25336,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:04.721959+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:02:04.721959+0000","last_peered":"2023-10-04T02:02:04.721959+0000","last_clean":"2023-10-04T02:02:04.721959+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:02:04.721959+0000","last_undegraded":"2023-10-04T02:02:04.721959+0000","last_fullsized":"2023-10-04T02:02:04.721959+0000","mapping_epoch":12,"log_start":"97'9400","ondisk_log_start":"97'9400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":342,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.568513+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:23.568513+0000","last_peered":"2023-10-04T02:01:23.568513+0000","last_clean":"2023-10-04T02:01:23.568513+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:23.568513+0000","last_undegraded":"2023-10-04T02:01:23.568513+0000","last_fullsized":"2023-10-04T02:01:23.568513+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29018349587,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2250001,"num_read_kb":8872800,"num_write":2245516,"num_write_kb":8982058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18397761536,"data_stored":38084857894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40285,"ondisk_log_size":40285,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61057,"ondisk_log_size":61057,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739437,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10654876,"kb_used_data":8986388,"kb_used_omap":2779,"kb_used_meta":1665700,"kb_avail":83716964,"statfs":{"total":96636764160,"available":85726171136,"internally_reserved":0,"allocated":9202061312,"data_stored":19044279081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":1705677773},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.48699999999999999},"min":{"1min":0.18099999999999999,"5min":0.17100000000000001,"15min":0.158},"max":{"1min":3.069,"5min":4.665,"15min":6.7199999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.50800000000000001,"15min":0.54400000000000004},"min":{"1min":0.246,"5min":0.222,"15min":0.20899999999999999},"max":{"1min":3.1539999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.5,"15min":0.59699999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51100000000000001,"15min":0.57799999999999996},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.1859999999999999,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.27300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739435,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6447756,"kb_used_data":4418308,"kb_used_omap":1193,"kb_used_meta":2028246,"kb_avail":87924084,"statfs":{"total":96636764160,"available":90034262016,"internally_reserved":0,"allocated":4524347392,"data_stored":9374062777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":2076924650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,4,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.307,"15min":0.313},"min":{"1min":0.192,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.54600000000000004,"5min":0.86199999999999999,"15min":3.1850000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.47199999999999998,"15min":0.53300000000000003},"min":{"1min":0.23200000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.57599999999999996,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.25900000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:01:11 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.192,"5min":0.183,"15min":0.17299999999999999},"max":{"1min":0.52400000000000002,"5min":4.4219999999999997,"15min":5.9790000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.23200000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":5.1870000000000003,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.20599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6494808,"kb_used_data":4568208,"kb_used_omap":1358,"kb_used_meta":1925233,"kb_avail":87877032,"statfs":{"total":96636764160,"available":89986080768,"internally_reserved":0,"allocated":4677844992,"data_stored":9668365328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":1971438828},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.79000000000000004,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.76100000000000001,"5min":4.851,"15min":5.6210000000000004},"last":0.85699999999999998}]},{"osd":2,"last update":"Wed Oct 4 02:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.867,"5min":5.4450000000000003,"15min":26.321999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51500000000000001,"15min":0.439},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":4.4489999999999998,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4675887104,"data_stored":9667862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4521746432,"data_stored":9373319187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9200128000,"data_stored":19043676179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:02:10.022 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:02:10.022 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:02:26.482 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:02:26.462+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:02:30.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:02:30.498 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:02:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:02:30.526 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2837,"stamp":"2023-10-04T02:02:29.875400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29052011065,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2469226,"num_read_kb":9081741,"num_write":2309134,"num_write_kb":9070075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101467,"ondisk_log_size":101467,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23909540,"kb_used_data":18036620,"kb_used_omap":5331,"kb_used_meta":5867564,"kb_avail":259205980,"statfs":{"total":289910292480,"available":265426923520,"internally_reserved":0,"allocated":18469498880,"data_stored":38151952370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5459037,"internal_metadata":6008386467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,4,3,10,3,11,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20684800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5156,"num_read_kb":20540,"num_write":4987,"num_write_kb":19948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002175"},"pg_stats":[{"pgid":"2.7","version":"160'9256","reported_seq":12981,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.694239+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:02:25.694239+0000","last_peered":"2023-10-04T02:02:25.694239+0000","last_clean":"2023-10-04T02:02:25.694239+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:02:25.694239+0000","last_undegraded":"2023-10-04T02:02:25.694239+0000","last_fullsized":"2023-10-04T02:02:25.694239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'7885","reported_seq":11758,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.684147+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:02:25.684147+0000","last_peered":"2023-10-04T02:02:25.684147+0000","last_clean":"2023-10-04T02:02:25.684147+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:02:25.684147+0000","last_undegraded":"2023-10-04T02:02:25.684147+0000","last_fullsized":"2023-10-04T02:02:25.684147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'9405","reported_seq":13438,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.694231+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:02:25.694231+0000","last_peered":"2023-10-04T02:02:25.694231+0000","last_clean":"2023-10-04T02:02:25.694231+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:02:25.694231+0000","last_undegraded":"2023-10-04T02:02:25.694231+0000","last_fullsized":"2023-10-04T02:02:25.694231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'7835","reported_seq":12714,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.688633+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:02:25.688633+0000","last_peered":"2023-10-04T02:02:25.688633+0000","last_clean":"2023-10-04T02:02:25.688633+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:02:25.688633+0000","last_undegraded":"2023-10-04T02:02:25.688633+0000","last_fullsized":"2023-10-04T02:02:25.688633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"160'578313","reported_seq":1141196,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:28.820392+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:02:28.820392+0000","last_peered":"2023-10-04T02:02:28.820392+0000","last_clean":"2023-10-04T02:02:28.820392+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:02:28.820392+0000","last_undegraded":"2023-10-04T02:02:28.820392+0000","last_fullsized":"2023-10-04T02:02:28.820392+0000","mapping_epoch":15,"log_start":"159'568273","ondisk_log_start":"159'568273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7297089536,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":567774,"num_read_kb":2238900,"num_write":566764,"num_write_kb":2267056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"160'18629","reported_seq":24309,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:28.618745+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:02:28.618745+0000","last_peered":"2023-10-04T02:02:28.618745+0000","last_clean":"2023-10-04T02:02:28.618745+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:02:28.618745+0000","last_undegraded":"2023-10-04T02:02:28.618745+0000","last_fullsized":"2023-10-04T02:02:28.618745+0000","mapping_epoch":12,"log_start":"95'8600","ondisk_log_start":"95'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"160'565130","reported_seq":1115398,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.912778+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:02:25.912778+0000","last_peered":"2023-10-04T02:02:25.912778+0000","last_clean":"2023-10-04T02:02:25.912778+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:02:25.912778+0000","last_undegraded":"2023-10-04T02:02:25.912778+0000","last_fullsized":"2023-10-04T02:02:25.912778+0000","mapping_epoch":15,"log_start":"159'555067","ondisk_log_start":"159'555067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7142002688,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":555084,"num_read_kb":2188872,"num_write":553750,"num_write_kb":2215000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"156'2103","reported_seq":9719,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:28.638623+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:02:28.638623+0000","last_peered":"2023-10-04T02:02:28.638623+0000","last_clean":"2023-10-04T02:02:28.638623+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:02:28.638623+0000","last_undegraded":"2023-10-04T02:02:28.638623+0000","last_fullsized":"2023-10-04T02:02:28.638623+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"160'590711","reported_seq":1166225,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:28.816106+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:02:28.816106+0000","last_peered":"2023-10-04T02:02:28.816106+0000","last_clean":"2023-10-04T02:02:28.816106+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:02:28.816106+0000","last_undegraded":"2023-10-04T02:02:28.816106+0000","last_fullsized":"2023-10-04T02:02:28.816106+0000","mapping_epoch":15,"log_start":"159'580666","ondisk_log_start":"159'580666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7472791552,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":580516,"num_read_kb":2289948,"num_write":578850,"num_write_kb":2315400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"156'4535","reported_seq":10095,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:28.621781+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:02:28.621781+0000","last_peered":"2023-10-04T02:02:28.621781+0000","last_clean":"2023-10-04T02:02:28.621781+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:02:28.621781+0000","last_undegraded":"2023-10-04T02:02:28.621781+0000","last_fullsized":"2023-10-04T02:02:28.621781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'563958","reported_seq":1113943,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:02:25.916969+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:02:25.916969+0000","last_peered":"2023-10-04T02:02:25.916969+0000","last_clean":"2023-10-04T02:02:25.916969+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:02:25.916969+0000","last_undegraded":"2023-10-04T02:02:25.916969+0000","last_fullsized":"2023-10-04T02:02:25.916969+0000","mapping_epoch":15,"log_start":"159'553863","ondisk_log_start":"159'553863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4102,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 570s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7139520531,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":554678,"num_read_kb":2187168,"num_write":554126,"num_write_kb":2216498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'19477","reported_seq":25381,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:02:25.698307+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:02:25.698307+0000","last_peered":"2023-10-04T02:02:25.698307+0000","last_clean":"2023-10-04T02:02:25.698307+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:02:25.698307+0000","last_undegraded":"2023-10-04T02:02:25.698307+0000","last_fullsized":"2023-10-04T02:02:25.698307+0000","mapping_epoch":12,"log_start":"97'9400","ondisk_log_start":"97'9400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":342,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:23.568513+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:23.568513+0000","last_peered":"2023-10-04T02:01:23.568513+0000","last_clean":"2023-10-04T02:01:23.568513+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:23.568513+0000","last_undegraded":"2023-10-04T02:01:23.568513+0000","last_fullsized":"2023-10-04T02:01:23.568513+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29051404307,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2258052,"num_read_kb":8904888,"num_write":2253490,"num_write_kb":9013954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18462928896,"data_stored":38150025254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61125,"ondisk_log_size":61125,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739441,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10808364,"kb_used_data":9017828,"kb_used_omap":2779,"kb_used_meta":1787748,"kb_avail":83563476,"statfs":{"total":96636764160,"available":85568999424,"internally_reserved":0,"allocated":9234255872,"data_stored":19076473641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":1830654925},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.44800000000000001,"15min":0.46400000000000002},"min":{"1min":0.192,"5min":0.18099999999999999,"15min":0.158},"max":{"1min":0.56499999999999995,"5min":4.665,"15min":6.7199999999999998},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.71499999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.5,"15min":0.59699999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51100000000000001,"15min":0.57799999999999996},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.1859999999999999,"5min":6.4249999999999998,"15min":35.423999999999999},"last":3.8370000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739439,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6500060,"kb_used_data":4434516,"kb_used_omap":1193,"kb_used_meta":2064342,"kb_avail":87871780,"statfs":{"total":96636764160,"available":89980702720,"internally_reserved":0,"allocated":4540944384,"data_stored":9390659769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":2113886954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,7,3,11,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.31900000000000001,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.52800000000000002},"min":{"1min":0.248,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.0880000000000001,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.53000000000000003},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":1.881,"5min":3.4159999999999999,"15min":5.9790000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.53,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359739439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6601116,"kb_used_data":4584276,"kb_used_omap":1358,"kb_used_meta":2015473,"kb_avail":87770724,"statfs":{"total":96636764160,"available":89877221376,"internally_reserved":0,"allocated":4694298624,"data_stored":9684818960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":2063844588},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.79000000000000004,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.76100000000000001,"5min":4.851,"15min":5.6210000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Wed Oct 4 02:02:15 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45200000000000001,"15min":0.54900000000000004},"min":{"1min":0.253,"5min":0.219,"15min":0.219},"max":{"1min":0.89200000000000002,"5min":3.867,"15min":26.321999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.47999999999999998,"15min":0.439},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3089999999999999,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4692439040,"data_stored":9684414464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4537864192,"data_stored":9389436947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9232625664,"data_stored":19076173843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:02:30.530 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:02:30.530 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:02:37.492 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:37.482+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:02:49.325 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.318+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:02:49.342 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.326+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:02:49.342 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.330+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:02:49.342 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.330+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.330+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63cbf32b0 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:02:49.343 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:02:49.334+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T02:02:50.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:02:50.960 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:02:50.961 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:02:50.984 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2848,"stamp":"2023-10-04T02:02:49.878735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29085061689,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2477187,"num_read_kb":9113509,"num_write":2317139,"num_write_kb":9102095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101437,"ondisk_log_size":101437,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23990496,"kb_used_data":18100680,"kb_used_omap":5331,"kb_used_meta":5884460,"kb_avail":259125024,"statfs":{"total":289910292480,"available":265344024576,"internally_reserved":0,"allocated":18535096320,"data_stored":38217536865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5459037,"internal_metadata":6025687971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,3,1,9,12,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20987904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4985,"num_read_kb":19884,"num_write":5088,"num_write_kb":20352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001996"},"pg_stats":[{"pgid":"2.7","version":"161'9276","reported_seq":13009,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.983208+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:02:43.983208+0000","last_peered":"2023-10-04T02:02:43.983208+0000","last_clean":"2023-10-04T02:02:43.983208+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:02:43.983208+0000","last_undegraded":"2023-10-04T02:02:43.983208+0000","last_fullsized":"2023-10-04T02:02:43.983208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'7895","reported_seq":11773,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.167837+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:02:43.167837+0000","last_peered":"2023-10-04T02:02:43.167837+0000","last_clean":"2023-10-04T02:02:43.167837+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:02:43.167837+0000","last_undegraded":"2023-10-04T02:02:43.167837+0000","last_fullsized":"2023-10-04T02:02:43.167837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7895,"log_dups_size":0,"ondisk_log_size":7895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'9426","reported_seq":13468,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.982813+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:02:43.982813+0000","last_peered":"2023-10-04T02:02:43.982813+0000","last_clean":"2023-10-04T02:02:43.982813+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:02:43.982813+0000","last_undegraded":"2023-10-04T02:02:43.982813+0000","last_fullsized":"2023-10-04T02:02:43.982813+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'7855","reported_seq":12743,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.983003+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:02:43.983003+0000","last_peered":"2023-10-04T02:02:43.983003+0000","last_clean":"2023-10-04T02:02:43.983003+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:02:43.983003+0000","last_undegraded":"2023-10-04T02:02:43.983003+0000","last_fullsized":"2023-10-04T02:02:43.983003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7855,"log_dups_size":0,"ondisk_log_size":7855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"161'580322","reported_seq":1145177,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:48.881484+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:02:48.881484+0000","last_peered":"2023-10-04T02:02:48.881484+0000","last_clean":"2023-10-04T02:02:48.881484+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:02:48.881484+0000","last_undegraded":"2023-10-04T02:02:48.881484+0000","last_fullsized":"2023-10-04T02:02:48.881484+0000","mapping_epoch":15,"log_start":"159'570273","ondisk_log_start":"159'570273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7305355264,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":569748,"num_read_kb":2246784,"num_write":568773,"num_write_kb":2275092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"161'18683","reported_seq":24373,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:48.762468+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:02:48.762468+0000","last_peered":"2023-10-04T02:02:48.762468+0000","last_clean":"2023-10-04T02:02:48.762468+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:02:48.762468+0000","last_undegraded":"2023-10-04T02:02:48.762468+0000","last_fullsized":"2023-10-04T02:02:48.762468+0000","mapping_epoch":12,"log_start":"95'8600","ondisk_log_start":"95'8600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"161'567169","reported_seq":1119375,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:46.040921+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:02:46.040921+0000","last_peered":"2023-10-04T02:02:46.040921+0000","last_clean":"2023-10-04T02:02:46.040921+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:02:46.040921+0000","last_undegraded":"2023-10-04T02:02:46.040921+0000","last_fullsized":"2023-10-04T02:02:46.040921+0000","mapping_epoch":15,"log_start":"159'557167","ondisk_log_start":"159'557167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7150383104,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":557019,"num_read_kb":2196584,"num_write":555789,"num_write_kb":2223156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9732,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.985963+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:02:43.985963+0000","last_peered":"2023-10-04T02:02:43.985963+0000","last_clean":"2023-10-04T02:02:43.985963+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:02:43.985963+0000","last_undegraded":"2023-10-04T02:02:43.985963+0000","last_fullsized":"2023-10-04T02:02:43.985963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"161'592720","reported_seq":1170304,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:48.873067+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:02:48.873067+0000","last_peered":"2023-10-04T02:02:48.873067+0000","last_clean":"2023-10-04T02:02:48.873067+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:02:48.873067+0000","last_undegraded":"2023-10-04T02:02:48.873067+0000","last_fullsized":"2023-10-04T02:02:48.873067+0000","mapping_epoch":15,"log_start":"159'582666","ondisk_log_start":"159'582666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7481077760,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":582584,"num_read_kb":2298208,"num_write":580859,"num_write_kb":2323436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10107,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.990868+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:02:43.990868+0000","last_peered":"2023-10-04T02:02:43.990868+0000","last_clean":"2023-10-04T02:02:43.990868+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:02:43.990868+0000","last_undegraded":"2023-10-04T02:02:43.990868+0000","last_fullsized":"2023-10-04T02:02:43.990868+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"161'566008","reported_seq":1117992,"reported_epoch":161,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:02:46.046769+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:02:46.046769+0000","last_peered":"2023-10-04T02:02:46.046769+0000","last_clean":"2023-10-04T02:02:46.046769+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:02:46.046769+0000","last_undegraded":"2023-10-04T02:02:46.046769+0000","last_fullsized":"2023-10-04T02:02:46.046769+0000","mapping_epoch":15,"log_start":"159'555963","ondisk_log_start":"159'555963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4228,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 590s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7147638803,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":556662,"num_read_kb":2195080,"num_write":556074,"num_write_kb":2224290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"161'19508","reported_seq":25425,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:02:43.984097+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:02:43.984097+0000","last_peered":"2023-10-04T02:02:43.984097+0000","last_clean":"2023-10-04T02:02:43.984097+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:02:43.984097+0000","last_undegraded":"2023-10-04T02:02:43.984097+0000","last_fullsized":"2023-10-04T02:02:43.984097+0000","mapping_epoch":12,"log_start":"97'9500","ondisk_log_start":"97'9500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":343,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:45.753085+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:45.753085+0000","last_peered":"2023-10-04T02:01:45.753085+0000","last_clean":"2023-10-04T02:01:45.753085+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:45.753085+0000","last_undegraded":"2023-10-04T02:01:45.753085+0000","last_fullsized":"2023-10-04T02:01:45.753085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29084454931,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2266013,"num_read_kb":8936656,"num_write":2261495,"num_write_kb":9045974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18528608256,"data_stored":38215704614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40150,"ondisk_log_size":40150,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61188,"ondisk_log_size":61188,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10976884,"kb_used_data":9049708,"kb_used_omap":2779,"kb_used_meta":1924388,"kb_avail":83394956,"statfs":{"total":96636764160,"available":85396434944,"internally_reserved":0,"allocated":9266900992,"data_stored":19109114446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":1970574285},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.44800000000000001,"15min":0.46400000000000002},"min":{"1min":0.192,"5min":0.18099999999999999,"15min":0.158},"max":{"1min":0.56499999999999995,"5min":4.665,"15min":6.7199999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.71499999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.753}]},{"osd":1,"last update":"Wed Oct 4 02:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.5,"15min":0.59699999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51100000000000001,"15min":0.57799999999999996},"min":{"1min":0.23599999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.1859999999999999,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.57599999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739443,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6334956,"kb_used_data":4449828,"kb_used_omap":1193,"kb_used_meta":1883926,"kb_avail":88036884,"statfs":{"total":96636764160,"available":90149769216,"internally_reserved":0,"allocated":4556623872,"data_stored":9406334942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":1929140970},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,9,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.31900000000000001,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.52800000000000002},"min":{"1min":0.248,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.0880000000000001,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.434}]},{"osd":2,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.53000000000000003},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":1.881,"5min":3.4159999999999999,"15min":5.9790000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.53,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.39300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6678656,"kb_used_data":4601144,"kb_used_omap":1358,"kb_used_meta":2076145,"kb_avail":87693184,"statfs":{"total":96636764160,"available":89797820416,"internally_reserved":0,"allocated":4711571456,"data_stored":9702087477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1391380,"internal_metadata":2125972716},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.79000000000000004,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":2.9369999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":0.76100000000000001,"5min":4.851,"15min":5.6210000000000004},"last":2.883}]},{"osd":2,"last update":"Wed Oct 4 02:02:15 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45200000000000001,"15min":0.54900000000000004},"min":{"1min":0.253,"5min":0.219,"15min":0.219},"max":{"1min":0.89200000000000002,"5min":3.867,"15min":26.321999999999999},"last":2.9870000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.47999999999999998,"15min":0.439},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3089999999999999,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":4.3490000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4708892672,"data_stored":9700868096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4554186752,"data_stored":9405759507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9265528832,"data_stored":19109077011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:02:50.986 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:02:50.986 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:03:06.955 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:06.945+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:03:06.956 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:06.949+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:03:06.956 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:06.949+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:03:06.957 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:06.953+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:03:06.957 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:06.953+0000 7f29e174e640 -1 reset not still connected to 0x55a63d79b520 2023-10-04T02:03:10.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:03:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:03:11.468 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:03:11.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2858,"stamp":"2023-10-04T02:03:09.882180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29117243961,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2484873,"num_read_kb":9144181,"num_write":2324944,"num_write_kb":9133315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101617,"ondisk_log_size":101617,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24085348,"kb_used_data":18163468,"kb_used_omap":5409,"kb_used_meta":5916446,"kb_avail":259030172,"statfs":{"total":289910292480,"available":265246896128,"internally_reserved":0,"allocated":18599391232,"data_stored":38281831777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5539194,"internal_metadata":6058441350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,10,4,10,12,20],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20316160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4932,"num_read_kb":19680,"num_write":4925,"num_write_kb":19700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002073"},"pg_stats":[{"pgid":"2.7","version":"161'9304","reported_seq":13045,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:05.808037+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:03:05.808037+0000","last_peered":"2023-10-04T02:03:05.808037+0000","last_clean":"2023-10-04T02:03:05.808037+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:03:05.808037+0000","last_undegraded":"2023-10-04T02:03:05.808037+0000","last_fullsized":"2023-10-04T02:03:05.808037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'7909","reported_seq":11791,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:04.542987+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:03:04.542987+0000","last_peered":"2023-10-04T02:03:04.542987+0000","last_clean":"2023-10-04T02:03:04.542987+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:03:04.542987+0000","last_undegraded":"2023-10-04T02:03:04.542987+0000","last_fullsized":"2023-10-04T02:03:04.542987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'9454","reported_seq":13504,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:05.807868+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:03:05.807868+0000","last_peered":"2023-10-04T02:03:05.807868+0000","last_clean":"2023-10-04T02:03:05.807868+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:03:05.807868+0000","last_undegraded":"2023-10-04T02:03:05.807868+0000","last_fullsized":"2023-10-04T02:03:05.807868+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'7883","reported_seq":12779,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:05.807717+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:03:05.807717+0000","last_peered":"2023-10-04T02:03:05.807717+0000","last_clean":"2023-10-04T02:03:05.807717+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:03:05.807717+0000","last_undegraded":"2023-10-04T02:03:05.807717+0000","last_fullsized":"2023-10-04T02:03:05.807717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"161'582325","reported_seq":1149122,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:08.936547+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:03:08.936547+0000","last_peered":"2023-10-04T02:03:08.936547+0000","last_clean":"2023-10-04T02:03:08.936547+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:03:08.936547+0000","last_undegraded":"2023-10-04T02:03:08.936547+0000","last_fullsized":"2023-10-04T02:03:08.936547+0000","mapping_epoch":15,"log_start":"160'572273","ondisk_log_start":"160'572273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7313707008,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":571687,"num_read_kb":2254512,"num_write":570776,"num_write_kb":2283104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"161'18710","reported_seq":24412,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:08.801633+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:03:08.801633+0000","last_peered":"2023-10-04T02:03:08.801633+0000","last_clean":"2023-10-04T02:03:08.801633+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:03:08.801633+0000","last_undegraded":"2023-10-04T02:03:08.801633+0000","last_fullsized":"2023-10-04T02:03:08.801633+0000","mapping_epoch":12,"log_start":"97'8700","ondisk_log_start":"97'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"161'569049","reported_seq":1123083,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:06.133757+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:03:06.133757+0000","last_peered":"2023-10-04T02:03:06.133757+0000","last_clean":"2023-10-04T02:03:06.133757+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:03:06.133757+0000","last_undegraded":"2023-10-04T02:03:06.133757+0000","last_fullsized":"2023-10-04T02:03:06.133757+0000","mapping_epoch":15,"log_start":"159'558967","ondisk_log_start":"159'558967","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7158128640,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":558846,"num_read_kb":2203872,"num_write":557669,"num_write_kb":2230676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9749,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:04.486666+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:03:04.486666+0000","last_peered":"2023-10-04T02:03:04.486666+0000","last_clean":"2023-10-04T02:03:04.486666+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:03:04.486666+0000","last_undegraded":"2023-10-04T02:03:04.486666+0000","last_fullsized":"2023-10-04T02:03:04.486666+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"161'594702","reported_seq":1174274,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:08.929785+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:03:08.929785+0000","last_peered":"2023-10-04T02:03:08.929785+0000","last_clean":"2023-10-04T02:03:08.929785+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:03:08.929785+0000","last_undegraded":"2023-10-04T02:03:08.929785+0000","last_fullsized":"2023-10-04T02:03:08.929785+0000","mapping_epoch":15,"log_start":"160'584666","ondisk_log_start":"160'584666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7489228800,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":584572,"num_read_kb":2306144,"num_write":582841,"num_write_kb":2331364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10119,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:04.459431+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:03:04.459431+0000","last_peered":"2023-10-04T02:03:04.459431+0000","last_clean":"2023-10-04T02:03:04.459431+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:03:04.459431+0000","last_undegraded":"2023-10-04T02:03:04.459431+0000","last_fullsized":"2023-10-04T02:03:04.459431+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"161'568056","reported_seq":1121982,"reported_epoch":161,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:03:06.148904+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:03:06.148904+0000","last_peered":"2023-10-04T02:03:06.148904+0000","last_clean":"2023-10-04T02:03:06.148904+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:03:06.148904+0000","last_undegraded":"2023-10-04T02:03:06.148904+0000","last_fullsized":"2023-10-04T02:03:06.148904+0000","mapping_epoch":15,"log_start":"160'557963","ondisk_log_start":"160'557963","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4354,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 610s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7155572755,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":558594,"num_read_kb":2202800,"num_write":558014,"num_write_kb":2232050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"161'19550","reported_seq":25479,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:05.834883+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:03:05.834883+0000","last_peered":"2023-10-04T02:03:05.834883+0000","last_clean":"2023-10-04T02:03:05.834883+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:03:05.834883+0000","last_undegraded":"2023-10-04T02:03:05.834883+0000","last_fullsized":"2023-10-04T02:03:05.834883+0000","mapping_epoch":12,"log_start":"97'9500","ondisk_log_start":"97'9500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":343,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:45.753085+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:45.753085+0000","last_peered":"2023-10-04T02:01:45.753085+0000","last_clean":"2023-10-04T02:01:45.753085+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:45.753085+0000","last_undegraded":"2023-10-04T02:01:45.753085+0000","last_fullsized":"2023-10-04T02:01:45.753085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29116637203,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2273699,"num_read_kb":8967328,"num_write":2269300,"num_write_kb":9077194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18592378880,"data_stored":38279475238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40263,"ondisk_log_size":40263,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61255,"ondisk_log_size":61255,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739449,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11093944,"kb_used_data":9080752,"kb_used_omap":2779,"kb_used_meta":2010404,"kb_avail":83277896,"statfs":{"total":96636764160,"available":85276565504,"internally_reserved":0,"allocated":9298690048,"data_stored":19140903502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":2058654669},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.44800000000000001,"15min":0.46400000000000002},"min":{"1min":0.192,"5min":0.18099999999999999,"15min":0.158},"max":{"1min":0.56499999999999995,"5min":4.665,"15min":6.7199999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.71499999999999997,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.47899999999999998,"15min":0.56599999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67800000000000005,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.8370000000000002,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.70599999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739447,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6404988,"kb_used_data":4465716,"kb_used_omap":1193,"kb_used_meta":1938070,"kb_avail":87966852,"statfs":{"total":96636764160,"available":90078056448,"internally_reserved":0,"allocated":4572893184,"data_stored":9422604254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":1984584426},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,1,3,5,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.31900000000000001,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.52800000000000002},"min":{"1min":0.248,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.0880000000000001,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:02:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.504,"15min":0.53000000000000003},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":1.881,"5min":3.4159999999999999,"15min":5.9790000000000001},"last":0.38},{"interface":"front","average":{"1min":0.373,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.53,"5min":5.1870000000000003,"15min":6.1050000000000004},"last":0.34300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6586416,"kb_used_data":4617000,"kb_used_omap":1437,"kb_used_meta":1967970,"kb_avail":87785424,"statfs":{"total":96636764160,"available":89892274176,"internally_reserved":0,"allocated":4727808000,"data_stored":9718324021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":2015202255},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,7,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.47299999999999998,"15min":0.437},"min":{"1min":0.20599999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.6789999999999998,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.51500000000000001},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":2.883,"5min":4.851,"15min":5.6210000000000004},"last":0.45900000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:02:15 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45200000000000001,"15min":0.54900000000000004},"min":{"1min":0.253,"5min":0.219,"15min":0.219},"max":{"1min":0.89200000000000002,"5min":3.867,"15min":26.321999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.47999999999999998,"15min":0.439},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3089999999999999,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4725219328,"data_stored":9717194752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4569841664,"data_stored":9421414419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9297317888,"data_stored":19140866067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:03:11.498 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:03:11.498 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:03:31.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:03:32.069 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:03:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:03:32.095 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2869,"stamp":"2023-10-04T02:03:31.886145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29153194553,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2493393,"num_read_kb":9178177,"num_write":2333728,"num_write_kb":9168451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101808,"ondisk_log_size":101808,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24395944,"kb_used_data":18229452,"kb_used_omap":5409,"kb_used_meta":6161054,"kb_avail":258719576,"statfs":{"total":289910292480,"available":264928845824,"internally_reserved":0,"allocated":18666958848,"data_stored":38349399393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5539194,"internal_metadata":6308919942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,6,15,18,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20045824,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4725,"num_read_kb":18848,"num_write":4901,"num_write_kb":19604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002004"},"pg_stats":[{"pgid":"2.7","version":"161'9334","reported_seq":13085,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:30.596928+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:03:30.596928+0000","last_peered":"2023-10-04T02:03:30.596928+0000","last_clean":"2023-10-04T02:03:30.596928+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:03:30.596928+0000","last_undegraded":"2023-10-04T02:03:30.596928+0000","last_fullsized":"2023-10-04T02:03:30.596928+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9334,"log_dups_size":0,"ondisk_log_size":9334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'7924","reported_seq":11811,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:30.064032+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:03:30.064032+0000","last_peered":"2023-10-04T02:03:30.064032+0000","last_clean":"2023-10-04T02:03:30.064032+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:03:30.064032+0000","last_undegraded":"2023-10-04T02:03:30.064032+0000","last_fullsized":"2023-10-04T02:03:30.064032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'9484","reported_seq":13545,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:30.597077+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:03:30.597077+0000","last_peered":"2023-10-04T02:03:30.597077+0000","last_clean":"2023-10-04T02:03:30.597077+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:03:30.597077+0000","last_undegraded":"2023-10-04T02:03:30.597077+0000","last_fullsized":"2023-10-04T02:03:30.597077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9484,"log_dups_size":0,"ondisk_log_size":9484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'7913","reported_seq":12820,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:30.585314+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:03:30.585314+0000","last_peered":"2023-10-04T02:03:30.585314+0000","last_clean":"2023-10-04T02:03:30.585314+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:03:30.585314+0000","last_undegraded":"2023-10-04T02:03:30.585314+0000","last_fullsized":"2023-10-04T02:03:30.585314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"161'584305","reported_seq":1153050,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:29.087809+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:03:29.087809+0000","last_peered":"2023-10-04T02:03:29.087809+0000","last_clean":"2023-10-04T02:03:29.087809+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:03:29.087809+0000","last_undegraded":"2023-10-04T02:03:29.087809+0000","last_fullsized":"2023-10-04T02:03:29.087809+0000","mapping_epoch":15,"log_start":"160'574273","ondisk_log_start":"160'574273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7321755648,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":573637,"num_read_kb":2262288,"num_write":572756,"num_write_kb":2291024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"161'18752","reported_seq":24467,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:28.189731+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:03:28.189731+0000","last_peered":"2023-10-04T02:03:28.189731+0000","last_clean":"2023-10-04T02:03:28.189731+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:03:28.189731+0000","last_undegraded":"2023-10-04T02:03:28.189731+0000","last_fullsized":"2023-10-04T02:03:28.189731+0000","mapping_epoch":12,"log_start":"97'8700","ondisk_log_start":"97'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"161'571452","reported_seq":1127791,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:31.257599+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:03:31.257599+0000","last_peered":"2023-10-04T02:03:31.257599+0000","last_clean":"2023-10-04T02:03:31.257599+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:03:31.257599+0000","last_undegraded":"2023-10-04T02:03:31.257599+0000","last_fullsized":"2023-10-04T02:03:31.257599+0000","mapping_epoch":15,"log_start":"160'561367","ondisk_log_start":"160'561367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7167987712,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":561151,"num_read_kb":2213080,"num_write":560072,"num_write_kb":2240288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9765,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:24.947609+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:03:24.947609+0000","last_peered":"2023-10-04T02:03:24.947609+0000","last_clean":"2023-10-04T02:03:24.947609+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:03:24.947609+0000","last_undegraded":"2023-10-04T02:03:24.947609+0000","last_fullsized":"2023-10-04T02:03:24.947609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"161'596718","reported_seq":1178232,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:29.087586+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:03:29.087586+0000","last_peered":"2023-10-04T02:03:29.087586+0000","last_clean":"2023-10-04T02:03:29.087586+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:03:29.087586+0000","last_undegraded":"2023-10-04T02:03:29.087586+0000","last_fullsized":"2023-10-04T02:03:29.087586+0000","mapping_epoch":15,"log_start":"160'586666","ondisk_log_start":"160'586666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7497494528,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":586514,"num_read_kb":2313892,"num_write":584857,"num_write_kb":2339428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10131,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:24.939191+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:03:24.939191+0000","last_peered":"2023-10-04T02:03:24.939191+0000","last_clean":"2023-10-04T02:03:24.939191+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:03:24.939191+0000","last_undegraded":"2023-10-04T02:03:24.939191+0000","last_fullsized":"2023-10-04T02:03:24.939191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"161'570558","reported_seq":1126819,"reported_epoch":161,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:03:31.258336+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:03:31.258336+0000","last_peered":"2023-10-04T02:03:31.258336+0000","last_clean":"2023-10-04T02:03:31.258336+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:03:31.258336+0000","last_undegraded":"2023-10-04T02:03:31.258336+0000","last_fullsized":"2023-10-04T02:03:31.258336+0000","mapping_epoch":15,"log_start":"160'560463","ondisk_log_start":"160'560463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4497,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 635s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7165349907,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":560917,"num_read_kb":2212064,"num_write":560399,"num_write_kb":2241590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"161'19593","reported_seq":25538,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-04T02:03:30.606316+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:03:30.606316+0000","last_peered":"2023-10-04T02:03:30.606316+0000","last_clean":"2023-10-04T02:03:30.606316+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:03:30.606316+0000","last_undegraded":"2023-10-04T02:03:30.606316+0000","last_fullsized":"2023-10-04T02:03:30.606316+0000","mapping_epoch":12,"log_start":"97'9500","ondisk_log_start":"97'9500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":343,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-04T02:01:45.753085+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:01:45.753085+0000","last_peered":"2023-10-04T02:01:45.753085+0000","last_clean":"2023-10-04T02:01:45.753085+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:01:45.753085+0000","last_undegraded":"2023-10-04T02:01:45.753085+0000","last_fullsized":"2023-10-04T02:01:45.753085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29152587795,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2282219,"num_read_kb":9001324,"num_write":2278084,"num_write_kb":9112330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18660155392,"data_stored":38347251750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61445,"ondisk_log_size":61445,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739453,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11237592,"kb_used_data":9111692,"kb_used_omap":2779,"kb_used_meta":2123108,"kb_avail":83134248,"statfs":{"total":96636764160,"available":85129469952,"internally_reserved":0,"allocated":9330372608,"data_stored":19172586062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2845747,"internal_metadata":2174063565},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.46000000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.8170000000000002,"5min":4.665,"15min":6.7199999999999998},"last":2.4489999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.8519999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":2.484}]},{"osd":1,"last update":"Wed Oct 4 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.47899999999999998,"15min":0.56599999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67800000000000005,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.8370000000000002,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739452,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6495492,"kb_used_data":4484796,"kb_used_omap":1193,"kb_used_meta":2009494,"kb_avail":87876348,"statfs":{"total":96636764160,"available":89985380352,"internally_reserved":0,"allocated":4592431104,"data_stored":9442142174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1221910,"internal_metadata":2057722602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.223,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.499,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.49099999999999999,"15min":0.53800000000000003},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.4199999999999999,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.59799999999999998,"15min":0.55900000000000005},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":8.5760000000000005,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.60599999999999998,"15min":0.54900000000000004},"min":{"1min":0.254,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.4770000000000003,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6662860,"kb_used_data":4632964,"kb_used_omap":1437,"kb_used_meta":2028450,"kb_avail":87708980,"statfs":{"total":96636764160,"available":89813995520,"internally_reserved":0,"allocated":4744155136,"data_stored":9734671157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":2077133775},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.47299999999999998,"15min":0.437},"min":{"1min":0.20599999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.6789999999999998,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.51500000000000001},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":2.883,"5min":4.851,"15min":5.6210000000000004},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Oct 4 02:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.56000000000000005},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":2.9870000000000001,"5min":3.867,"15min":26.321999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54100000000000004,"15min":0.45200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.3490000000000002,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4741582848,"data_stored":9733558272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4589445120,"data_stored":9441017875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9329127424,"data_stored":19172675603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:03:32.098 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:03:32.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T02:03:32.558 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T02:03:32.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:03:52.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:03:53.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:03:53.059 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:03:53.087 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2880,"stamp":"2023-10-04T02:03:51.891292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29185524281,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2501139,"num_read_kb":9209081,"num_write":2341600,"num_write_kb":9199939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101737,"ondisk_log_size":101737,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23526696,"kb_used_data":18292368,"kb_used_omap":5652,"kb_used_meta":5228651,"kb_avail":259588824,"statfs":{"total":289910292480,"available":265818955776,"internally_reserved":0,"allocated":18731384832,"data_stored":38413812303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":5354139138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,4,2,13,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20140032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4844,"num_read_kb":19332,"num_write":4908,"num_write_kb":19632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003837"},"pg_stats":[{"pgid":"2.7","version":"162'9360","reported_seq":13120,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:51.277565+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:03:51.277565+0000","last_peered":"2023-10-04T02:03:51.277565+0000","last_clean":"2023-10-04T02:03:51.277565+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:03:51.277565+0000","last_undegraded":"2023-10-04T02:03:51.277565+0000","last_fullsized":"2023-10-04T02:03:51.277565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"162'7937","reported_seq":11829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:50.543128+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:03:50.543128+0000","last_peered":"2023-10-04T02:03:50.543128+0000","last_clean":"2023-10-04T02:03:50.543128+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:03:50.543128+0000","last_undegraded":"2023-10-04T02:03:50.543128+0000","last_fullsized":"2023-10-04T02:03:50.543128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"162'9510","reported_seq":13580,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:51.277835+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:03:51.277835+0000","last_peered":"2023-10-04T02:03:51.277835+0000","last_clean":"2023-10-04T02:03:51.277835+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:03:51.277835+0000","last_undegraded":"2023-10-04T02:03:51.277835+0000","last_fullsized":"2023-10-04T02:03:51.277835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9510,"log_dups_size":0,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"162'7939","reported_seq":12855,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:51.276902+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:03:51.276902+0000","last_peered":"2023-10-04T02:03:51.276902+0000","last_clean":"2023-10-04T02:03:51.276902+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:03:51.276902+0000","last_undegraded":"2023-10-04T02:03:51.276902+0000","last_fullsized":"2023-10-04T02:03:51.276902+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7939,"log_dups_size":0,"ondisk_log_size":7939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"162'586317","reported_seq":1157023,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:49.158669+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:03:49.158669+0000","last_peered":"2023-10-04T02:03:49.158669+0000","last_clean":"2023-10-04T02:03:49.158669+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:03:49.158669+0000","last_undegraded":"2023-10-04T02:03:49.158669+0000","last_fullsized":"2023-10-04T02:03:49.158669+0000","mapping_epoch":15,"log_start":"160'576273","ondisk_log_start":"160'576273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7330013184,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":575595,"num_read_kb":2270092,"num_write":574768,"num_write_kb":2299072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"162'18764","reported_seq":24492,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:47.051083+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:03:47.051083+0000","last_peered":"2023-10-04T02:03:47.051083+0000","last_clean":"2023-10-04T02:03:47.051083+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:03:47.051083+0000","last_undegraded":"2023-10-04T02:03:47.051083+0000","last_fullsized":"2023-10-04T02:03:47.051083+0000","mapping_epoch":12,"log_start":"97'8700","ondisk_log_start":"97'8700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"162'573401","reported_seq":1131607,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:51.330825+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:03:51.330825+0000","last_peered":"2023-10-04T02:03:51.330825+0000","last_clean":"2023-10-04T02:03:51.330825+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:03:51.330825+0000","last_undegraded":"2023-10-04T02:03:51.330825+0000","last_fullsized":"2023-10-04T02:03:51.330825+0000","mapping_epoch":15,"log_start":"160'563367","ondisk_log_start":"160'563367","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7176024064,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":563015,"num_read_kb":2220512,"num_write":562021,"num_write_kb":2248084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9782,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:45.424263+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:03:45.424263+0000","last_peered":"2023-10-04T02:03:45.424263+0000","last_clean":"2023-10-04T02:03:45.424263+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:03:45.424263+0000","last_undegraded":"2023-10-04T02:03:45.424263+0000","last_fullsized":"2023-10-04T02:03:45.424263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"162'598735","reported_seq":1182223,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:49.151976+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:03:49.151976+0000","last_peered":"2023-10-04T02:03:49.151976+0000","last_clean":"2023-10-04T02:03:49.151976+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:03:49.151976+0000","last_undegraded":"2023-10-04T02:03:49.151976+0000","last_fullsized":"2023-10-04T02:03:49.151976+0000","mapping_epoch":15,"log_start":"160'588666","ondisk_log_start":"160'588666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7505756160,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":588487,"num_read_kb":2321780,"num_write":586874,"num_write_kb":2347496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10145,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:45.424750+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:03:45.424750+0000","last_peered":"2023-10-04T02:03:45.424750+0000","last_clean":"2023-10-04T02:03:45.424750+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:03:45.424750+0000","last_undegraded":"2023-10-04T02:03:45.424750+0000","last_fullsized":"2023-10-04T02:03:45.424750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"162'572567","reported_seq":1130793,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:03:51.362824+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:03:51.362824+0000","last_peered":"2023-10-04T02:03:51.362824+0000","last_clean":"2023-10-04T02:03:51.362824+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:03:51.362824+0000","last_undegraded":"2023-10-04T02:03:51.362824+0000","last_fullsized":"2023-10-04T02:03:51.362824+0000","mapping_epoch":15,"log_start":"160'562563","ondisk_log_start":"160'562563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4640,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 655s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7173124115,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":562868,"num_read_kb":2219844,"num_write":562293,"num_write_kb":2249166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"162'19632","reported_seq":25590,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:51.313420+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:03:51.313420+0000","last_peered":"2023-10-04T02:03:51.313420+0000","last_clean":"2023-10-04T02:03:51.313420+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:03:51.313420+0000","last_undegraded":"2023-10-04T02:03:51.313420+0000","last_fullsized":"2023-10-04T02:03:51.313420+0000","mapping_epoch":12,"log_start":"99'9600","ondisk_log_start":"99'9600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":345,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:38.595912+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:38.595912+0000","last_peered":"2023-10-04T02:03:38.595912+0000","last_clean":"2023-10-04T02:03:38.595912+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:38.595912+0000","last_undegraded":"2023-10-04T02:03:38.595912+0000","last_fullsized":"2023-10-04T02:03:38.595912+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29184917523,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2289965,"num_read_kb":9032228,"num_write":2285956,"num_write_kb":9143818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18724896768,"data_stored":38411993126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40151,"ondisk_log_size":40151,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61487,"ondisk_log_size":61487,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739457,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10390704,"kb_used_data":9143016,"kb_used_omap":2858,"kb_used_meta":1244821,"kb_avail":83981136,"statfs":{"total":96636764160,"available":85996683264,"internally_reserved":0,"allocated":9362448384,"data_stored":19204657480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1274697339},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.46000000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.8170000000000002,"5min":4.665,"15min":6.7199999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.8519999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:02:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.47899999999999998,"15min":0.56599999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67800000000000005,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.8370000000000002,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.41699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739456,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6402896,"kb_used_data":4500296,"kb_used_omap":1357,"kb_used_meta":1901234,"kb_avail":87968944,"statfs":{"total":96636764160,"available":90080198656,"internally_reserved":0,"allocated":4608303104,"data_stored":9458009816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1946864568},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,1,6,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.223,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.499,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.49099999999999999,"15min":0.53800000000000003},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.4199999999999999,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.32800000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.59799999999999998,"15min":0.55900000000000005},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":8.5760000000000005,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.60599999999999998,"15min":0.54900000000000004},"min":{"1min":0.254,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.4770000000000003,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6733096,"kb_used_data":4649056,"kb_used_omap":1437,"kb_used_meta":2082594,"kb_avail":87638744,"statfs":{"total":96636764160,"available":89742073856,"internally_reserved":0,"allocated":4760633344,"data_stored":9751145007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":2132577231},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.47299999999999998,"15min":0.437},"min":{"1min":0.20599999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.6789999999999998,"5min":3.8010000000000002,"15min":5.4850000000000003},"last":5.1230000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.51500000000000001},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.21099999999999999},"max":{"1min":2.883,"5min":4.851,"15min":5.6210000000000004},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Oct 4 02:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.56000000000000005},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":2.9870000000000001,"5min":3.867,"15min":26.321999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54100000000000004,"15min":0.45200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.3490000000000002,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4758097920,"data_stored":9750073344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4605194240,"data_stored":9456766995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9361604608,"data_stored":19205152787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:03:53.089 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:03:53.089 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:03:55.842 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:03:55.832+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T02:03:55.843 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:03:55.832+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:03:57.410 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:57.404+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:03:57.422 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:57.412+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:03:57.422 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:57.416+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:03:57.422 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:03:57.416+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:04:03.029 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:04:03.030 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:04:03.030 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T02:04:03.030 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:04:03.031 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:04:03.031 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:03.020+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:04:13.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:04:13.628 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:04:13.629 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:04:13.653 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2890,"stamp":"2023-10-04T02:04:11.894841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29218050617,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2509005,"num_read_kb":9240473,"num_write":2349504,"num_write_kb":9231555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101923,"ondisk_log_size":101923,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22803824,"kb_used_data":18357080,"kb_used_omap":5652,"kb_used_meta":4441067,"kb_avail":260311696,"statfs":{"total":289910292480,"available":266559176704,"internally_reserved":0,"allocated":18797649920,"data_stored":38480077391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4547653122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,5,13,2,12,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20086784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4970,"num_read_kb":19828,"num_write":4884,"num_write_kb":19536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002163"},"pg_stats":[{"pgid":"2.7","version":"162'9387","reported_seq":13155,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:11.018872+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:04:11.018872+0000","last_peered":"2023-10-04T02:04:11.018872+0000","last_clean":"2023-10-04T02:04:11.018872+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:04:11.018872+0000","last_undegraded":"2023-10-04T02:04:11.018872+0000","last_fullsized":"2023-10-04T02:04:11.018872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9387,"log_dups_size":0,"ondisk_log_size":9387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"162'7952","reported_seq":11848,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:07.378575+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:04:07.378575+0000","last_peered":"2023-10-04T02:04:07.378575+0000","last_clean":"2023-10-04T02:04:07.378575+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:04:07.378575+0000","last_undegraded":"2023-10-04T02:04:07.378575+0000","last_fullsized":"2023-10-04T02:04:07.378575+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"162'9539","reported_seq":13617,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:11.018636+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:04:11.018636+0000","last_peered":"2023-10-04T02:04:11.018636+0000","last_clean":"2023-10-04T02:04:11.018636+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:04:11.018636+0000","last_undegraded":"2023-10-04T02:04:11.018636+0000","last_fullsized":"2023-10-04T02:04:11.018636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9539,"log_dups_size":0,"ondisk_log_size":9539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"162'7968","reported_seq":12892,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:11.018674+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:04:11.018674+0000","last_peered":"2023-10-04T02:04:11.018674+0000","last_clean":"2023-10-04T02:04:11.018674+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:04:11.018674+0000","last_undegraded":"2023-10-04T02:04:11.018674+0000","last_fullsized":"2023-10-04T02:04:11.018674+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"162'588352","reported_seq":1161055,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:09.213019+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:04:09.213019+0000","last_peered":"2023-10-04T02:04:09.213019+0000","last_clean":"2023-10-04T02:04:09.213019+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:04:09.213019+0000","last_undegraded":"2023-10-04T02:04:09.213019+0000","last_fullsized":"2023-10-04T02:04:09.213019+0000","mapping_epoch":15,"log_start":"160'578273","ondisk_log_start":"160'578273","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7338422272,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":577591,"num_read_kb":2278068,"num_write":576803,"num_write_kb":2307212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"162'18811","reported_seq":24551,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:08.262380+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:04:08.262380+0000","last_peered":"2023-10-04T02:04:08.262380+0000","last_clean":"2023-10-04T02:04:08.262380+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:04:08.262380+0000","last_undegraded":"2023-10-04T02:04:08.262380+0000","last_fullsized":"2023-10-04T02:04:08.262380+0000","mapping_epoch":12,"log_start":"99'8800","ondisk_log_start":"99'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"162'575332","reported_seq":1135450,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:11.492799+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:04:11.492799+0000","last_peered":"2023-10-04T02:04:11.492799+0000","last_clean":"2023-10-04T02:04:11.492799+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:04:11.492799+0000","last_undegraded":"2023-10-04T02:04:11.492799+0000","last_fullsized":"2023-10-04T02:04:11.492799+0000","mapping_epoch":15,"log_start":"160'565267","ondisk_log_start":"160'565267","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7183937536,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":564927,"num_read_kb":2228156,"num_write":563952,"num_write_kb":2255808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9798,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:05.905330+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:04:05.905330+0000","last_peered":"2023-10-04T02:04:05.905330+0000","last_clean":"2023-10-04T02:04:05.905330+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:04:05.905330+0000","last_undegraded":"2023-10-04T02:04:05.905330+0000","last_fullsized":"2023-10-04T02:04:05.905330+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"162'600757","reported_seq":1186301,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:09.205128+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:04:09.205128+0000","last_peered":"2023-10-04T02:04:09.205128+0000","last_clean":"2023-10-04T02:04:09.205128+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:04:09.205128+0000","last_undegraded":"2023-10-04T02:04:09.205128+0000","last_fullsized":"2023-10-04T02:04:09.205128+0000","mapping_epoch":15,"log_start":"160'590666","ondisk_log_start":"160'590666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7514091520,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":590542,"num_read_kb":2329972,"num_write":588896,"num_write_kb":2355584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10157,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:05.899733+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:04:05.899733+0000","last_peered":"2023-10-04T02:04:05.899733+0000","last_clean":"2023-10-04T02:04:05.899733+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:04:05.899733+0000","last_undegraded":"2023-10-04T02:04:05.899733+0000","last_fullsized":"2023-10-04T02:04:05.899733+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"162'574577","reported_seq":1134702,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:04:11.500081+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:04:11.500081+0000","last_peered":"2023-10-04T02:04:11.500081+0000","last_clean":"2023-10-04T02:04:11.500081+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:04:11.500081+0000","last_undegraded":"2023-10-04T02:04:11.500081+0000","last_fullsized":"2023-10-04T02:04:11.500081+0000","mapping_epoch":15,"log_start":"160'564563","ondisk_log_start":"160'564563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4756,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 676s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7180992531,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":564771,"num_read_kb":2227424,"num_write":564209,"num_write_kb":2256830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"162'19673","reported_seq":25643,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:11.019201+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:04:11.019201+0000","last_peered":"2023-10-04T02:04:11.019201+0000","last_clean":"2023-10-04T02:04:11.019201+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:04:11.019201+0000","last_undegraded":"2023-10-04T02:04:11.019201+0000","last_fullsized":"2023-10-04T02:04:11.019201+0000","mapping_epoch":12,"log_start":"99'9600","ondisk_log_start":"99'9600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":345,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:38.595912+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:38.595912+0000","last_peered":"2023-10-04T02:03:38.595912+0000","last_clean":"2023-10-04T02:03:38.595912+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:38.595912+0000","last_undegraded":"2023-10-04T02:03:38.595912+0000","last_fullsized":"2023-10-04T02:03:38.595912+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29217443859,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2297831,"num_read_kb":9063620,"num_write":2293860,"num_write_kb":9175434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18791559168,"data_stored":38478655526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40249,"ondisk_log_size":40249,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61575,"ondisk_log_size":61575,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739462,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10514108,"kb_used_data":9176180,"kb_used_omap":2858,"kb_used_meta":1335061,"kb_avail":83857732,"statfs":{"total":96636764160,"available":85870317568,"internally_reserved":0,"allocated":9396408320,"data_stored":19238617416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1367103099},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.46000000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.8170000000000002,"5min":4.665,"15min":6.7199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.8519999999999999,"5min":4.7569999999999997,"15min":6.7930000000000001},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Oct 4 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.43099999999999999,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.1719999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70599999999999996,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.65500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739460,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6478852,"kb_used_data":4515708,"kb_used_omap":1357,"kb_used_meta":1961778,"kb_avail":87892988,"statfs":{"total":96636764160,"available":90002419712,"internally_reserved":0,"allocated":4624084992,"data_stored":9473791704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":2008861624},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2,12,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.223,"5min":0.189,"15min":0.17399999999999999},"max":{"1min":0.499,"5min":2.0459999999999998,"15min":3.1850000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.49099999999999999,"15min":0.53800000000000003},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":2.4199999999999999,"5min":5.8440000000000003,"15min":6.7169999999999996},"last":0.309}]},{"osd":2,"last update":"Wed Oct 4 02:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.59799999999999998,"15min":0.55900000000000005},"min":{"1min":0.20799999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":8.5760000000000005,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.60599999999999998,"15min":0.54900000000000004},"min":{"1min":0.254,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.4770000000000003,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.38900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5810864,"kb_used_data":4665192,"kb_used_omap":1437,"kb_used_meta":1144226,"kb_avail":88560976,"statfs":{"total":96636764160,"available":90686439424,"internally_reserved":0,"allocated":4777156608,"data_stored":9767668271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1171688399},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:03:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53000000000000003,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.52900000000000003,"15min":0.51100000000000001},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.54200000000000004,"5min":4.851,"15min":5.6210000000000004},"last":0.495}]},{"osd":2,"last update":"Wed Oct 4 02:03:21 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48499999999999999,"15min":0.56000000000000005},"min":{"1min":0.26600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":2.9870000000000001,"5min":3.867,"15min":26.321999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54100000000000004,"15min":0.45200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.3490000000000002,"5min":4.6340000000000003,"15min":5.2309999999999999},"last":0.88}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4774707200,"data_stored":9766682624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4620943360,"data_stored":9472516115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9395908608,"data_stored":19239456787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:04:13.657 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:04:13.657 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:04:16.944 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:16.940+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:04:16.948 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:16.940+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:04:16.948 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:16.940+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:04:33.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:04:34.169 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:04:34.170 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:04:34.200 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2901,"stamp":"2023-10-04T02:04:33.900381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29250228793,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2516652,"num_read_kb":9270985,"num_write":2357349,"num_write_kb":9262935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101908,"ondisk_log_size":101908,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22975164,"kb_used_data":18420644,"kb_used_omap":5652,"kb_used_meta":4548843,"kb_avail":260140356,"statfs":{"total":289910292480,"available":266383724544,"internally_reserved":0,"allocated":18862739456,"data_stored":38545166927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4658015746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,2,5,10,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16154624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3937,"num_read_kb":15720,"num_write":3939,"num_write_kb":15756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003792"},"pg_stats":[{"pgid":"2.7","version":"162'9415","reported_seq":13191,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:31.498916+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:04:31.498916+0000","last_peered":"2023-10-04T02:04:31.498916+0000","last_clean":"2023-10-04T02:04:31.498916+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:04:31.498916+0000","last_undegraded":"2023-10-04T02:04:31.498916+0000","last_fullsized":"2023-10-04T02:04:31.498916+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"162'7967","reported_seq":11867,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:30.714355+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:04:30.714355+0000","last_peered":"2023-10-04T02:04:30.714355+0000","last_clean":"2023-10-04T02:04:30.714355+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:04:30.714355+0000","last_undegraded":"2023-10-04T02:04:30.714355+0000","last_fullsized":"2023-10-04T02:04:30.714355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"162'9568","reported_seq":13654,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:31.498565+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:04:31.498565+0000","last_peered":"2023-10-04T02:04:31.498565+0000","last_clean":"2023-10-04T02:04:31.498565+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:04:31.498565+0000","last_undegraded":"2023-10-04T02:04:31.498565+0000","last_fullsized":"2023-10-04T02:04:31.498565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9568,"log_dups_size":0,"ondisk_log_size":9568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"162'7997","reported_seq":12929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:31.498590+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:04:31.498590+0000","last_peered":"2023-10-04T02:04:31.498590+0000","last_clean":"2023-10-04T02:04:31.498590+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:04:31.498590+0000","last_undegraded":"2023-10-04T02:04:31.498590+0000","last_fullsized":"2023-10-04T02:04:31.498590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"162'590383","reported_seq":1164967,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:29.334097+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:04:29.334097+0000","last_peered":"2023-10-04T02:04:29.334097+0000","last_clean":"2023-10-04T02:04:29.334097+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:04:29.334097+0000","last_undegraded":"2023-10-04T02:04:29.334097+0000","last_fullsized":"2023-10-04T02:04:29.334097+0000","mapping_epoch":15,"log_start":"161'580373","ondisk_log_start":"161'580373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7346749440,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":579475,"num_read_kb":2285588,"num_write":578834,"num_write_kb":2315336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"162'18847","reported_seq":24599,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:26.379136+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:04:26.379136+0000","last_peered":"2023-10-04T02:04:26.379136+0000","last_clean":"2023-10-04T02:04:26.379136+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:04:26.379136+0000","last_undegraded":"2023-10-04T02:04:26.379136+0000","last_fullsized":"2023-10-04T02:04:26.379136+0000","mapping_epoch":12,"log_start":"99'8800","ondisk_log_start":"99'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"162'577241","reported_seq":1139245,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:31.619036+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:04:31.619036+0000","last_peered":"2023-10-04T02:04:31.619036+0000","last_clean":"2023-10-04T02:04:31.619036+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:04:31.619036+0000","last_undegraded":"2023-10-04T02:04:31.619036+0000","last_fullsized":"2023-10-04T02:04:31.619036+0000","mapping_epoch":15,"log_start":"161'567167","ondisk_log_start":"161'567167","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7191797760,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":566817,"num_read_kb":2235696,"num_write":565861,"num_write_kb":2263444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9814,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:26.408124+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:04:26.408124+0000","last_peered":"2023-10-04T02:04:26.408124+0000","last_clean":"2023-10-04T02:04:26.408124+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:04:26.408124+0000","last_undegraded":"2023-10-04T02:04:26.408124+0000","last_fullsized":"2023-10-04T02:04:26.408124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"162'602767","reported_seq":1190345,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:29.337326+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:04:29.337326+0000","last_peered":"2023-10-04T02:04:29.337326+0000","last_clean":"2023-10-04T02:04:29.337326+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:04:29.337326+0000","last_undegraded":"2023-10-04T02:04:29.337326+0000","last_fullsized":"2023-10-04T02:04:29.337326+0000","mapping_epoch":15,"log_start":"161'592766","ondisk_log_start":"161'592766","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7522320384,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":592576,"num_read_kb":2338092,"num_write":590906,"num_write_kb":2363624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10169,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:26.393390+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:04:26.393390+0000","last_peered":"2023-10-04T02:04:26.393390+0000","last_clean":"2023-10-04T02:04:26.393390+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:04:26.393390+0000","last_undegraded":"2023-10-04T02:04:26.393390+0000","last_fullsized":"2023-10-04T02:04:26.393390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"162'576633","reported_seq":1138629,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:04:31.641709+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:04:31.641709+0000","last_peered":"2023-10-04T02:04:31.641709+0000","last_clean":"2023-10-04T02:04:31.641709+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:04:31.641709+0000","last_undegraded":"2023-10-04T02:04:31.641709+0000","last_fullsized":"2023-10-04T02:04:31.641709+0000","mapping_epoch":15,"log_start":"161'566563","ondisk_log_start":"161'566563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":4953,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 696s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7188754451,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":566610,"num_read_kb":2234756,"num_write":566104,"num_write_kb":2264410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"162'19715","reported_seq":25697,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:31.498837+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:04:31.498837+0000","last_peered":"2023-10-04T02:04:31.498837+0000","last_clean":"2023-10-04T02:04:31.498837+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:04:31.498837+0000","last_undegraded":"2023-10-04T02:04:31.498837+0000","last_fullsized":"2023-10-04T02:04:31.498837+0000","mapping_epoch":12,"log_start":"101'9700","ondisk_log_start":"101'9700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":345,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:38.595912+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:38.595912+0000","last_peered":"2023-10-04T02:03:38.595912+0000","last_clean":"2023-10-04T02:03:38.595912+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:38.595912+0000","last_undegraded":"2023-10-04T02:03:38.595912+0000","last_fullsized":"2023-10-04T02:03:38.595912+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29249622035,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2305478,"num_read_kb":9094132,"num_write":2301705,"num_write_kb":9206814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18855665664,"data_stored":38542762022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40155,"ondisk_log_size":40155,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61654,"ondisk_log_size":61654,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10582148,"kb_used_data":9208636,"kb_used_omap":2858,"kb_used_meta":1370645,"kb_avail":83789692,"statfs":{"total":96636764160,"available":85800644608,"internally_reserved":0,"allocated":9429643264,"data_stored":19271852360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1403541115},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.48499999999999999,"15min":0.45800000000000002},"min":{"1min":0.17599999999999999,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.4660000000000002,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.216},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.55100000000000005,"15min":0.51600000000000001},"min":{"1min":0.26300000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.532,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Wed Oct 4 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.43099999999999999,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.1719999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.247},{"interface":"front","average":{"1min":0.378,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70599999999999996,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.19900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739464,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6530144,"kb_used_data":4530904,"kb_used_omap":1357,"kb_used_meta":1997874,"kb_avail":87841696,"statfs":{"total":96636764160,"available":89949896704,"internally_reserved":0,"allocated":4639645696,"data_stored":9489352408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":2045823928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.187,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.67100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":8.7409999999999997,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.53000000000000003,"15min":0.52700000000000002},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":0.70999999999999996,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.746,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.32700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5862872,"kb_used_data":4681104,"kb_used_omap":1437,"kb_used_meta":1180322,"kb_avail":88508968,"statfs":{"total":96636764160,"available":90633183232,"internally_reserved":0,"allocated":4793450496,"data_stored":9783962159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1208650703},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:03:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53000000000000003,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.52900000000000003,"15min":0.51100000000000001},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.54200000000000004,"5min":4.851,"15min":5.6210000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:04:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.57199999999999995},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.219},"max":{"1min":3.5019999999999998,"5min":3.867,"15min":26.321999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.46400000000000002},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":4.9770000000000003,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4791263232,"data_stored":9783238656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4636528640,"data_stored":9488101395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9427873792,"data_stored":19271421971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:04:34.202 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:04:34.202 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:04:40.692 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:40.687+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:04:40.698 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:40.691+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:04:40.699 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:40.691+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:04:40.699 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:04:40.691+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:04:54.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:04:54.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:04:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:04:54.838 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2912,"stamp":"2023-10-04T02:04:53.903825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29281661497,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2524075,"num_read_kb":9300585,"num_write":2365046,"num_write_kb":9293723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102141,"ondisk_log_size":102141,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22040068,"kb_used_data":18482024,"kb_used_omap":5652,"kb_used_meta":3552363,"kb_avail":261075452,"statfs":{"total":289910292480,"available":267341262848,"internally_reserved":0,"allocated":18925592576,"data_stored":38608006844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":3637620226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,8,7,17,23,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15966208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3751,"num_read_kb":14952,"num_write":3906,"num_write_kb":15624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001711"},"pg_stats":[{"pgid":"2.7","version":"163'9449","reported_seq":13233,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:50.077400+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:04:50.077400+0000","last_peered":"2023-10-04T02:04:50.077400+0000","last_clean":"2023-10-04T02:04:50.077400+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:04:50.077400+0000","last_undegraded":"2023-10-04T02:04:50.077400+0000","last_fullsized":"2023-10-04T02:04:50.077400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"163'7985","reported_seq":11890,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:49.902502+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:04:49.902502+0000","last_peered":"2023-10-04T02:04:49.902502+0000","last_clean":"2023-10-04T02:04:49.902502+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:04:49.902502+0000","last_undegraded":"2023-10-04T02:04:49.902502+0000","last_fullsized":"2023-10-04T02:04:49.902502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'9603","reported_seq":13696,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:50.065044+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:04:50.065044+0000","last_peered":"2023-10-04T02:04:50.065044+0000","last_clean":"2023-10-04T02:04:50.065044+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:04:50.065044+0000","last_undegraded":"2023-10-04T02:04:50.065044+0000","last_fullsized":"2023-10-04T02:04:50.065044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'8032","reported_seq":12971,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:50.073287+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:04:50.073287+0000","last_peered":"2023-10-04T02:04:50.073287+0000","last_clean":"2023-10-04T02:04:50.073287+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:04:50.073287+0000","last_undegraded":"2023-10-04T02:04:50.073287+0000","last_fullsized":"2023-10-04T02:04:50.073287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8032,"log_dups_size":0,"ondisk_log_size":8032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"163'592374","reported_seq":1168761,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:49.404018+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:04:49.404018+0000","last_peered":"2023-10-04T02:04:49.404018+0000","last_clean":"2023-10-04T02:04:49.404018+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:04:49.404018+0000","last_undegraded":"2023-10-04T02:04:49.404018+0000","last_fullsized":"2023-10-04T02:04:49.404018+0000","mapping_epoch":15,"log_start":"161'582373","ondisk_log_start":"161'582373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7354880000,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":581275,"num_read_kb":2292752,"num_write":580825,"num_write_kb":2323300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"163'18883","reported_seq":24648,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:49.248991+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:04:49.248991+0000","last_peered":"2023-10-04T02:04:49.248991+0000","last_clean":"2023-10-04T02:04:49.248991+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:04:49.248991+0000","last_undegraded":"2023-10-04T02:04:49.248991+0000","last_fullsized":"2023-10-04T02:04:49.248991+0000","mapping_epoch":12,"log_start":"99'8800","ondisk_log_start":"99'8800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"163'579088","reported_seq":1142922,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:51.726604+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:04:51.726604+0000","last_peered":"2023-10-04T02:04:51.726604+0000","last_clean":"2023-10-04T02:04:51.726604+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:04:51.726604+0000","last_undegraded":"2023-10-04T02:04:51.726604+0000","last_fullsized":"2023-10-04T02:04:51.726604+0000","mapping_epoch":15,"log_start":"161'569067","ondisk_log_start":"161'569067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7199342592,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":568653,"num_read_kb":2243016,"num_write":567708,"num_write_kb":2270832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9831,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:46.859561+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:04:46.859561+0000","last_peered":"2023-10-04T02:04:46.859561+0000","last_clean":"2023-10-04T02:04:46.859561+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:04:46.859561+0000","last_undegraded":"2023-10-04T02:04:46.859561+0000","last_fullsized":"2023-10-04T02:04:46.859561+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"163'604731","reported_seq":1194186,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:49.391383+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:04:49.391383+0000","last_peered":"2023-10-04T02:04:49.391383+0000","last_clean":"2023-10-04T02:04:49.391383+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:04:49.391383+0000","last_undegraded":"2023-10-04T02:04:49.391383+0000","last_fullsized":"2023-10-04T02:04:49.391383+0000","mapping_epoch":15,"log_start":"161'594666","ondisk_log_start":"161'594666","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7530328064,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":594454,"num_read_kb":2345580,"num_write":592870,"num_write_kb":2371480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10182,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:04:46.863854+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:04:46.863854+0000","last_peered":"2023-10-04T02:04:46.863854+0000","last_clean":"2023-10-04T02:04:46.863854+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:04:46.863854+0000","last_undegraded":"2023-10-04T02:04:46.863854+0000","last_fullsized":"2023-10-04T02:04:46.863854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"163'578654","reported_seq":1142574,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:04:51.746145+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:04:51.746145+0000","last_peered":"2023-10-04T02:04:51.746145+0000","last_clean":"2023-10-04T02:04:51.746145+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:04:51.746145+0000","last_undegraded":"2023-10-04T02:04:51.746145+0000","last_fullsized":"2023-10-04T02:04:51.746145+0000","mapping_epoch":15,"log_start":"161'568563","ondisk_log_start":"161'568563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5109,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 716s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7196504083,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":568519,"num_read_kb":2242384,"num_write":567999,"num_write_kb":2271990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"163'19767","reported_seq":25759,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:50.102688+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:04:50.102688+0000","last_peered":"2023-10-04T02:04:50.102688+0000","last_clean":"2023-10-04T02:04:50.102688+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:04:50.102688+0000","last_undegraded":"2023-10-04T02:04:50.102688+0000","last_fullsized":"2023-10-04T02:04:50.102688+0000","mapping_epoch":12,"log_start":"101'9700","ondisk_log_start":"101'9700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":346,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:46.316522+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:46.316522+0000","last_peered":"2023-10-04T02:03:46.316522+0000","last_clean":"2023-10-04T02:03:46.316522+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:46.316522+0000","last_undegraded":"2023-10-04T02:03:46.316522+0000","last_fullsized":"2023-10-04T02:03:46.316522+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29281054739,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2312901,"num_read_kb":9123732,"num_write":2309402,"num_write_kb":9237602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18918735872,"data_stored":38605832230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40178,"ondisk_log_size":40178,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61864,"ondisk_log_size":61864,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739470,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10698964,"kb_used_data":9239372,"kb_used_omap":2858,"kb_used_meta":1456725,"kb_avail":83672876,"statfs":{"total":96636764160,"available":85681025024,"internally_reserved":0,"allocated":9461116928,"data_stored":19303321623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1491687035},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.48499999999999999,"15min":0.45800000000000002},"min":{"1min":0.17599999999999999,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.4660000000000002,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.253},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.55100000000000005,"15min":0.51600000000000001},"min":{"1min":0.26300000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.532,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Oct 4 02:03:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.43099999999999999,"15min":0.55000000000000004},"min":{"1min":0.224,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.1719999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.378,"5min":0.53900000000000003,"15min":0.57199999999999995},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70599999999999996,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.36699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5408312,"kb_used_data":4545776,"kb_used_omap":1357,"kb_used_meta":861170,"kb_avail":88963528,"statfs":{"total":96636764160,"available":91098652672,"internally_reserved":0,"allocated":4654874624,"data_stored":9504576935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":881839032},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,4,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.187,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.67100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.371},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":8.7409999999999997,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.53000000000000003,"15min":0.52700000000000002},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":0.70999999999999996,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.312},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.746,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.35099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739467,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5932792,"kb_used_data":4696876,"kb_used_omap":1437,"kb_used_meta":1234466,"kb_avail":88439048,"statfs":{"total":96636764160,"available":90561585152,"internally_reserved":0,"allocated":4809601024,"data_stored":9800108286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1264094159},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,3,9,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:03:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53000000000000003,"15min":0.45300000000000001},"min":{"1min":0.19,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.52900000000000003,"15min":0.51100000000000001},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.54200000000000004,"5min":4.851,"15min":5.6210000000000004},"last":0.221}]},{"osd":2,"last update":"Wed Oct 4 02:04:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.57199999999999995},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.219},"max":{"1min":3.5019999999999998,"5min":3.867,"15min":26.321999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.46400000000000002},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":4.9770000000000003,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.36799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4807458816,"data_stored":9799434240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4651864064,"data_stored":9503436819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9459412992,"data_stored":19302961171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:04:54.841 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:04:54.842 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:05:04.488 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:05:04.479+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T02:05:04.488 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:05:04.479+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e1380bd110 2023-10-04T02:05:04.488 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:05:04.479+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:05:14.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:05:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:05:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:05:15.362 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2922,"stamp":"2023-10-04T02:05:13.909383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29314863673,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2531943,"num_read_kb":9331961,"num_write":2373159,"num_write_kb":9326175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102173,"ondisk_log_size":102173,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22319924,"kb_used_data":18546840,"kb_used_omap":5652,"kb_used_meta":3767403,"kb_avail":260795596,"statfs":{"total":289910292480,"available":267054690304,"internally_reserved":0,"allocated":18991964160,"data_stored":38674378428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":3857821186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,4,6,2,6,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16179200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3886,"num_read_kb":15492,"num_write":3951,"num_write_kb":15804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002004"},"pg_stats":[{"pgid":"2.7","version":"163'9471","reported_seq":13263,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.898883+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:05:09.898883+0000","last_peered":"2023-10-04T02:05:09.898883+0000","last_clean":"2023-10-04T02:05:09.898883+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:05:09.898883+0000","last_undegraded":"2023-10-04T02:05:09.898883+0000","last_fullsized":"2023-10-04T02:05:09.898883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"163'7996","reported_seq":11905,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:07.339233+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:05:07.339233+0000","last_peered":"2023-10-04T02:05:07.339233+0000","last_clean":"2023-10-04T02:05:07.339233+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:05:07.339233+0000","last_undegraded":"2023-10-04T02:05:07.339233+0000","last_fullsized":"2023-10-04T02:05:07.339233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":7996,"log_dups_size":0,"ondisk_log_size":7996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'9625","reported_seq":13726,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.890795+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:05:09.890795+0000","last_peered":"2023-10-04T02:05:09.890795+0000","last_clean":"2023-10-04T02:05:09.890795+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:05:09.890795+0000","last_undegraded":"2023-10-04T02:05:09.890795+0000","last_fullsized":"2023-10-04T02:05:09.890795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9625,"log_dups_size":0,"ondisk_log_size":9625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'8054","reported_seq":13001,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.889305+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:05:09.889305+0000","last_peered":"2023-10-04T02:05:09.889305+0000","last_clean":"2023-10-04T02:05:09.889305+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:05:09.889305+0000","last_undegraded":"2023-10-04T02:05:09.889305+0000","last_fullsized":"2023-10-04T02:05:09.889305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"163'594382","reported_seq":1172834,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.506253+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:05:09.506253+0000","last_peered":"2023-10-04T02:05:09.506253+0000","last_clean":"2023-10-04T02:05:09.506253+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:05:09.506253+0000","last_undegraded":"2023-10-04T02:05:09.506253+0000","last_fullsized":"2023-10-04T02:05:09.506253+0000","mapping_epoch":15,"log_start":"161'584373","ondisk_log_start":"161'584373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7363092480,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":583342,"num_read_kb":2300976,"num_write":582833,"num_write_kb":2331332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"163'18907","reported_seq":24684,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:07.748967+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:05:07.748967+0000","last_peered":"2023-10-04T02:05:07.748967+0000","last_clean":"2023-10-04T02:05:07.748967+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:05:07.748967+0000","last_undegraded":"2023-10-04T02:05:07.748967+0000","last_fullsized":"2023-10-04T02:05:07.748967+0000","mapping_epoch":12,"log_start":"101'8900","ondisk_log_start":"101'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"163'581110","reported_seq":1146819,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:11.843465+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:05:11.843465+0000","last_peered":"2023-10-04T02:05:11.843465+0000","last_clean":"2023-10-04T02:05:11.843465+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:05:11.843465+0000","last_undegraded":"2023-10-04T02:05:11.843465+0000","last_fullsized":"2023-10-04T02:05:11.843465+0000","mapping_epoch":15,"log_start":"161'571067","ondisk_log_start":"161'571067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7207632896,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":570518,"num_read_kb":2250456,"num_write":569730,"num_write_kb":2278920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9848,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:07.340876+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:05:07.340876+0000","last_peered":"2023-10-04T02:05:07.340876+0000","last_clean":"2023-10-04T02:05:07.340876+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:05:07.340876+0000","last_undegraded":"2023-10-04T02:05:07.340876+0000","last_fullsized":"2023-10-04T02:05:07.340876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"163'606821","reported_seq":1198273,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.501541+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:05:09.501541+0000","last_peered":"2023-10-04T02:05:09.501541+0000","last_clean":"2023-10-04T02:05:09.501541+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:05:09.501541+0000","last_undegraded":"2023-10-04T02:05:09.501541+0000","last_fullsized":"2023-10-04T02:05:09.501541+0000","mapping_epoch":15,"log_start":"161'596766","ondisk_log_start":"161'596766","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7538880512,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":596450,"num_read_kb":2353552,"num_write":594960,"num_write_kb":2379840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10194,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:07.340432+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:05:07.340432+0000","last_peered":"2023-10-04T02:05:07.340432+0000","last_clean":"2023-10-04T02:05:07.340432+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:05:07.340432+0000","last_undegraded":"2023-10-04T02:05:07.340432+0000","last_fullsized":"2023-10-04T02:05:07.340432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"163'580732","reported_seq":1146599,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:05:11.835132+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:05:11.835132+0000","last_peered":"2023-10-04T02:05:11.835132+0000","last_clean":"2023-10-04T02:05:11.835132+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:05:11.835132+0000","last_undegraded":"2023-10-04T02:05:11.835132+0000","last_fullsized":"2023-10-04T02:05:11.835132+0000","mapping_epoch":15,"log_start":"161'570663","ondisk_log_start":"161'570663","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5216,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 736s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7204651027,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":570459,"num_read_kb":2250124,"num_write":569992,"num_write_kb":2279962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"163'19800","reported_seq":25804,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:09.935213+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:05:09.935213+0000","last_peered":"2023-10-04T02:05:09.935213+0000","last_clean":"2023-10-04T02:05:09.935213+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:05:09.935213+0000","last_undegraded":"2023-10-04T02:05:09.935213+0000","last_fullsized":"2023-10-04T02:05:09.935213+0000","mapping_epoch":12,"log_start":"101'9700","ondisk_log_start":"101'9700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":346,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:46.316522+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:46.316522+0000","last_peered":"2023-10-04T02:03:46.316522+0000","last_clean":"2023-10-04T02:03:46.316522+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:46.316522+0000","last_undegraded":"2023-10-04T02:03:46.316522+0000","last_fullsized":"2023-10-04T02:03:46.316522+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29314256915,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2320769,"num_read_kb":9155108,"num_write":2317515,"num_write_kb":9270054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18985177088,"data_stored":38672273446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61898,"ondisk_log_size":61898,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739474,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10849704,"kb_used_data":9271900,"kb_used_omap":2858,"kb_used_meta":1574933,"kb_avail":83522136,"statfs":{"total":96636764160,"available":85526667264,"internally_reserved":0,"allocated":9494425600,"data_stored":19336630295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1612732027},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.48499999999999999,"15min":0.45800000000000002},"min":{"1min":0.17599999999999999,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":3.4660000000000002,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.55100000000000005,"15min":0.51600000000000001},"min":{"1min":0.26300000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":3.532,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.25600000000000001}]},{"osd":1,"last update":"Wed Oct 4 02:05:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.434,"15min":0.53700000000000003},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.61699999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.55100000000000005,"15min":0.57099999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65500000000000003,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359739472,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5460456,"kb_used_data":4561824,"kb_used_omap":1357,"kb_used_meta":897266,"kb_avail":88911384,"statfs":{"total":96636764160,"available":91045257216,"internally_reserved":0,"allocated":4671307776,"data_stored":9521010087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":918801336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,2,6,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.187,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.67100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.249},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.20300000000000001},"max":{"1min":8.7409999999999997,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.377}]},{"osd":2,"last update":"Wed Oct 4 02:04:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.53000000000000003,"15min":0.52700000000000002},"min":{"1min":0.21099999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":0.70999999999999996,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.746,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6009764,"kb_used_data":4713116,"kb_used_omap":1437,"kb_used_meta":1295202,"kb_avail":88362076,"statfs":{"total":96636764160,"available":90482765824,"internally_reserved":0,"allocated":4826230784,"data_stored":9816738046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1326287823},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:04:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.54200000000000004,"15min":0.45400000000000001},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.53500000000000003,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":2.1080000000000001,"5min":4.851,"15min":5.6210000000000004},"last":2.5819999999999999}]},{"osd":2,"last update":"Wed Oct 4 02:04:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52700000000000002,"15min":0.57199999999999995},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.219},"max":{"1min":3.5019999999999998,"5min":3.867,"15min":26.321999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.52700000000000002,"15min":0.46400000000000002},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":4.9770000000000003,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4824240128,"data_stored":9816215552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4668297216,"data_stored":9519869971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9492639744,"data_stored":19336187923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:05:15.364 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:05:15.364 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:05:26.220 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:05:26.214+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:05:34.722 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:05:34.706+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T02:05:34.723 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:05:34.706+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:05:35.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:05:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:05:35.826 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:05:35.857 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2932,"stamp":"2023-10-04T02:05:33.912713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29347619385,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2539767,"num_read_kb":9363149,"num_write":2381144,"num_write_kb":9358115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102328,"ondisk_log_size":102328,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22570748,"kb_used_data":18611364,"kb_used_omap":5652,"kb_used_meta":3953707,"kb_avail":260544772,"statfs":{"total":289910292480,"available":266797846528,"internally_reserved":0,"allocated":19058036736,"data_stored":38740451004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4048596482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,7,3,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16060416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3914,"num_read_kb":15604,"num_write":3914,"num_write_kb":15656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002070"},"pg_stats":[{"pgid":"2.7","version":"163'9495","reported_seq":13295,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.922255+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:05:29.922255+0000","last_peered":"2023-10-04T02:05:29.922255+0000","last_clean":"2023-10-04T02:05:29.922255+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:05:29.922255+0000","last_undegraded":"2023-10-04T02:05:29.922255+0000","last_fullsized":"2023-10-04T02:05:29.922255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"163'8008","reported_seq":11921,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:28.822567+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:05:28.822567+0000","last_peered":"2023-10-04T02:05:28.822567+0000","last_clean":"2023-10-04T02:05:28.822567+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:05:28.822567+0000","last_undegraded":"2023-10-04T02:05:28.822567+0000","last_fullsized":"2023-10-04T02:05:28.822567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'9650","reported_seq":13760,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.919439+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:05:29.919439+0000","last_peered":"2023-10-04T02:05:29.919439+0000","last_clean":"2023-10-04T02:05:29.919439+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:05:29.919439+0000","last_undegraded":"2023-10-04T02:05:29.919439+0000","last_fullsized":"2023-10-04T02:05:29.919439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'8079","reported_seq":13035,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.919063+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:05:29.919063+0000","last_peered":"2023-10-04T02:05:29.919063+0000","last_clean":"2023-10-04T02:05:29.919063+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:05:29.919063+0000","last_undegraded":"2023-10-04T02:05:29.919063+0000","last_fullsized":"2023-10-04T02:05:29.919063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8079,"log_dups_size":0,"ondisk_log_size":8079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"163'596462","reported_seq":1176957,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.655873+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:05:29.655873+0000","last_peered":"2023-10-04T02:05:29.655873+0000","last_clean":"2023-10-04T02:05:29.655873+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:05:29.655873+0000","last_undegraded":"2023-10-04T02:05:29.655873+0000","last_fullsized":"2023-10-04T02:05:29.655873+0000","mapping_epoch":15,"log_start":"162'586373","ondisk_log_start":"162'586373","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7371612160,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":585382,"num_read_kb":2309112,"num_write":584913,"num_write_kb":2339652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"163'18952","reported_seq":24741,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:28.771998+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:05:28.771998+0000","last_peered":"2023-10-04T02:05:28.771998+0000","last_clean":"2023-10-04T02:05:28.771998+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:05:28.771998+0000","last_undegraded":"2023-10-04T02:05:28.771998+0000","last_fullsized":"2023-10-04T02:05:28.771998+0000","mapping_epoch":12,"log_start":"101'8900","ondisk_log_start":"101'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"163'583040","reported_seq":1150646,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:31.880468+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:05:31.880468+0000","last_peered":"2023-10-04T02:05:31.880468+0000","last_clean":"2023-10-04T02:05:31.880468+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:05:31.880468+0000","last_undegraded":"2023-10-04T02:05:31.880468+0000","last_fullsized":"2023-10-04T02:05:31.880468+0000","mapping_epoch":15,"log_start":"162'572967","ondisk_log_start":"162'572967","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7215554560,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":572413,"num_read_kb":2258000,"num_write":571660,"num_write_kb":2286640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9864,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:27.826977+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:05:27.826977+0000","last_peered":"2023-10-04T02:05:27.826977+0000","last_clean":"2023-10-04T02:05:27.826977+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:05:27.826977+0000","last_undegraded":"2023-10-04T02:05:27.826977+0000","last_fullsized":"2023-10-04T02:05:27.826977+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"163'608823","reported_seq":1202285,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.663104+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:05:29.663104+0000","last_peered":"2023-10-04T02:05:29.663104+0000","last_clean":"2023-10-04T02:05:29.663104+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:05:29.663104+0000","last_undegraded":"2023-10-04T02:05:29.663104+0000","last_fullsized":"2023-10-04T02:05:29.663104+0000","mapping_epoch":15,"log_start":"162'598766","ondisk_log_start":"162'598766","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7547097088,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":598461,"num_read_kb":2361584,"num_write":596962,"num_write_kb":2387848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10206,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:27.832428+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:05:27.832428+0000","last_peered":"2023-10-04T02:05:27.832428+0000","last_clean":"2023-10-04T02:05:27.832428+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:05:27.832428+0000","last_undegraded":"2023-10-04T02:05:27.832428+0000","last_fullsized":"2023-10-04T02:05:27.832428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"163'582807","reported_seq":1150563,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:05:31.901048+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:05:31.901048+0000","last_peered":"2023-10-04T02:05:31.901048+0000","last_clean":"2023-10-04T02:05:31.901048+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:05:31.901048+0000","last_undegraded":"2023-10-04T02:05:31.901048+0000","last_fullsized":"2023-10-04T02:05:31.901048+0000","mapping_epoch":15,"log_start":"162'572763","ondisk_log_start":"162'572763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5344,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 756s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7212748819,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":572337,"num_read_kb":2257600,"num_write":571965,"num_write_kb":2287854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"163'19837","reported_seq":25854,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:29.934294+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:05:29.934294+0000","last_peered":"2023-10-04T02:05:29.934294+0000","last_clean":"2023-10-04T02:05:29.934294+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:05:29.934294+0000","last_undegraded":"2023-10-04T02:05:29.934294+0000","last_fullsized":"2023-10-04T02:05:29.934294+0000","mapping_epoch":12,"log_start":"103'9800","ondisk_log_start":"103'9800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":346,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-10-04T02:03:46.316522+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:03:46.316522+0000","last_peered":"2023-10-04T02:03:46.316522+0000","last_clean":"2023-10-04T02:03:46.316522+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:03:46.316522+0000","last_undegraded":"2023-10-04T02:03:46.316522+0000","last_fullsized":"2023-10-04T02:03:46.316522+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29347012627,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2328593,"num_read_kb":9186296,"num_write":2325500,"num_write_kb":9301994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19050393600,"data_stored":38737489958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40263,"ondisk_log_size":40263,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":61966,"ondisk_log_size":61966,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10977724,"kb_used_data":9303860,"kb_used_omap":2858,"kb_used_meta":1670997,"kb_avail":83394116,"statfs":{"total":96636764160,"available":85395574784,"internally_reserved":0,"allocated":9527152640,"data_stored":19369357335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1711101563},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.53000000000000003,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.57499999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:05:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.434,"15min":0.53700000000000003},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.61699999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.318},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.55100000000000005,"15min":0.57099999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65500000000000003,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739476,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5530168,"kb_used_data":4577392,"kb_used_omap":1357,"kb_used_meta":951410,"kb_avail":88841672,"statfs":{"total":96636764160,"available":90973872128,"internally_reserved":0,"allocated":4687249408,"data_stored":9536951719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":974244792},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,6,3,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32700000000000001,"15min":0.316},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.54100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.49299999999999999,"15min":0.53100000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":3.9900000000000002,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.59799999999999998,"15min":0.54500000000000004},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.7639999999999998,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.55500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739475,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6062856,"kb_used_data":4730112,"kb_used_omap":1437,"kb_used_meta":1331298,"kb_avail":88308984,"statfs":{"total":96636764160,"available":90428399616,"internally_reserved":0,"allocated":4843634688,"data_stored":9834141950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1363250127},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:04:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.54200000000000004,"15min":0.45400000000000001},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.53500000000000003,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.505},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":2.1080000000000001,"5min":4.851,"15min":5.6210000000000004},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed Oct 4 02:05:25 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53500000000000003,"15min":0.58199999999999996},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.219},"max":{"1min":1.9259999999999999,"5min":3.867,"15min":26.321999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.49299999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.1239999999999997,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4840960000,"data_stored":9832935424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4684283904,"data_stored":9535856659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9525149696,"data_stored":19368697875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:05:35.859 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:05:35.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-04T02:05:36.330 INFO:teuthology.orchestra.run.smithi027.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-04T02:05:36.366 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:05:56.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:05:56.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:05:56.922 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:05:56.951 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2943,"stamp":"2023-10-04T02:05:55.920941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29384745529,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2548709,"num_read_kb":9398849,"num_write":2390195,"num_write_kb":9394319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102326,"ondisk_log_size":102326,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22911824,"kb_used_data":18687160,"kb_used_omap":5652,"kb_used_meta":4218987,"kb_avail":260203696,"statfs":{"total":289910292480,"available":266448584704,"internally_reserved":0,"allocated":19135651840,"data_stored":38818066108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4320243202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,3,7,10,8,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20889600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5004,"num_read_kb":19976,"num_write":5094,"num_write_kb":20376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002093"},"pg_stats":[{"pgid":"2.7","version":"163'9519","reported_seq":13327,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:51.687386+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:05:51.687386+0000","last_peered":"2023-10-04T02:05:51.687386+0000","last_clean":"2023-10-04T02:05:51.687386+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:05:51.687386+0000","last_undegraded":"2023-10-04T02:05:51.687386+0000","last_fullsized":"2023-10-04T02:05:51.687386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9519,"log_dups_size":0,"ondisk_log_size":9519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"163'8020","reported_seq":11937,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:50.680221+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:05:50.680221+0000","last_peered":"2023-10-04T02:05:50.680221+0000","last_clean":"2023-10-04T02:05:50.680221+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:05:50.680221+0000","last_undegraded":"2023-10-04T02:05:50.680221+0000","last_fullsized":"2023-10-04T02:05:50.680221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'9674","reported_seq":13792,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:51.687782+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:05:51.687782+0000","last_peered":"2023-10-04T02:05:51.687782+0000","last_clean":"2023-10-04T02:05:51.687782+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:05:51.687782+0000","last_undegraded":"2023-10-04T02:05:51.687782+0000","last_fullsized":"2023-10-04T02:05:51.687782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9674,"log_dups_size":0,"ondisk_log_size":9674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"163'8103","reported_seq":13067,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:51.687352+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:05:51.687352+0000","last_peered":"2023-10-04T02:05:51.687352+0000","last_clean":"2023-10-04T02:05:51.687352+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:05:51.687352+0000","last_undegraded":"2023-10-04T02:05:51.687352+0000","last_fullsized":"2023-10-04T02:05:51.687352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"163'598994","reported_seq":1181968,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:54.822825+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:05:54.822825+0000","last_peered":"2023-10-04T02:05:54.822825+0000","last_clean":"2023-10-04T02:05:54.822825+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:05:54.822825+0000","last_undegraded":"2023-10-04T02:05:54.822825+0000","last_fullsized":"2023-10-04T02:05:54.822825+0000","mapping_epoch":15,"log_start":"162'588973","ondisk_log_start":"162'588973","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7381995520,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":587860,"num_read_kb":2319008,"num_write":587445,"num_write_kb":2349780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"163'18970","reported_seq":24775,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:53.419359+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:05:53.419359+0000","last_peered":"2023-10-04T02:05:53.419359+0000","last_clean":"2023-10-04T02:05:53.419359+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:05:53.419359+0000","last_undegraded":"2023-10-04T02:05:53.419359+0000","last_fullsized":"2023-10-04T02:05:53.419359+0000","mapping_epoch":12,"log_start":"101'8900","ondisk_log_start":"101'8900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"163'585104","reported_seq":1154619,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:51.978632+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:05:51.978632+0000","last_peered":"2023-10-04T02:05:51.978632+0000","last_clean":"2023-10-04T02:05:51.978632+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:05:51.978632+0000","last_undegraded":"2023-10-04T02:05:51.978632+0000","last_fullsized":"2023-10-04T02:05:51.978632+0000","mapping_epoch":15,"log_start":"162'575067","ondisk_log_start":"162'575067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7224016896,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":574324,"num_read_kb":2265628,"num_write":573724,"num_write_kb":2294896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9884,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:53.420151+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:05:53.420151+0000","last_peered":"2023-10-04T02:05:53.420151+0000","last_clean":"2023-10-04T02:05:53.420151+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:05:53.420151+0000","last_undegraded":"2023-10-04T02:05:53.420151+0000","last_fullsized":"2023-10-04T02:05:53.420151+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"163'611378","reported_seq":1207399,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:54.822883+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:05:54.822883+0000","last_peered":"2023-10-04T02:05:54.822883+0000","last_clean":"2023-10-04T02:05:54.822883+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:05:54.822883+0000","last_undegraded":"2023-10-04T02:05:54.822883+0000","last_fullsized":"2023-10-04T02:05:54.822883+0000","mapping_epoch":15,"log_start":"162'601366","ondisk_log_start":"162'601366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7557607424,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":601017,"num_read_kb":2371788,"num_write":599517,"num_write_kb":2398068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10222,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:53.420082+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:05:53.420082+0000","last_peered":"2023-10-04T02:05:53.420082+0000","last_clean":"2023-10-04T02:05:53.420082+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:05:53.420082+0000","last_undegraded":"2023-10-04T02:05:53.420082+0000","last_fullsized":"2023-10-04T02:05:53.420082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"163'584816","reported_seq":1154579,"reported_epoch":163,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:05:51.975508+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:05:51.975508+0000","last_peered":"2023-10-04T02:05:51.975508+0000","last_clean":"2023-10-04T02:05:51.975508+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:05:51.975508+0000","last_undegraded":"2023-10-04T02:05:51.975508+0000","last_fullsized":"2023-10-04T02:05:51.975508+0000","mapping_epoch":15,"log_start":"162'574763","ondisk_log_start":"162'574763","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5478,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 776s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7220518931,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":574334,"num_read_kb":2265572,"num_write":573865,"num_write_kb":2295454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"163'19873","reported_seq":25902,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:51.701836+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:05:51.701836+0000","last_peered":"2023-10-04T02:05:51.701836+0000","last_clean":"2023-10-04T02:05:51.701836+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:05:51.701836+0000","last_undegraded":"2023-10-04T02:05:51.701836+0000","last_fullsized":"2023-10-04T02:05:51.701836+0000","mapping_epoch":12,"log_start":"103'9800","ondisk_log_start":"103'9800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":347,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:04:48.645404+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:04:48.645404+0000","last_peered":"2023-10-04T02:04:48.645404+0000","last_clean":"2023-10-04T02:04:48.645404+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:04:48.645404+0000","last_undegraded":"2023-10-04T02:04:48.645404+0000","last_fullsized":"2023-10-04T02:04:48.645404+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29384138771,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2337535,"num_read_kb":9221996,"num_write":2334551,"num_write_kb":9338198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19128475648,"data_stored":38815572006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40123,"ondisk_log_size":40123,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":62104,"ondisk_log_size":62104,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739483,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11180700,"kb_used_data":9343828,"kb_used_omap":2858,"kb_used_meta":1834005,"kb_avail":83191140,"statfs":{"total":96636764160,"available":85187727360,"internally_reserved":0,"allocated":9568079872,"data_stored":19410284567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1878021755},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.53000000000000003,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.57499999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Wed Oct 4 02:05:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.434,"15min":0.53700000000000003},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.61699999999999999,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.55100000000000005,"15min":0.57099999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65500000000000003,"5min":6.4249999999999998,"15min":35.423999999999999},"last":0.30599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739480,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5581992,"kb_used_data":4593120,"kb_used_omap":1357,"kb_used_meta":987506,"kb_avail":88789848,"statfs":{"total":96636764160,"available":90920804352,"internally_reserved":0,"allocated":4703354880,"data_stored":9553057191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1011207096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,3,3,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32700000000000001,"15min":0.316},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.54100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.49299999999999999,"15min":0.53100000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":3.9900000000000002,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.59799999999999998,"15min":0.54500000000000004},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.7639999999999998,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.439}]}]},{"osd":0,"up_from":8,"seq":34359739481,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6149132,"kb_used_data":4750212,"kb_used_omap":1437,"kb_used_meta":1397474,"kb_avail":88222708,"statfs":{"total":96636764160,"available":90340052992,"internally_reserved":0,"allocated":4864217088,"data_stored":9854724350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1431014351},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:04:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.54200000000000004,"15min":0.45400000000000001},"min":{"1min":0.26000000000000001,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.53500000000000003,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.22},"max":{"1min":2.1080000000000001,"5min":4.851,"15min":5.6210000000000004},"last":0.309}]},{"osd":2,"last update":"Wed Oct 4 02:05:25 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53500000000000003,"15min":0.58199999999999996},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.219},"max":{"1min":1.9259999999999999,"5min":3.867,"15min":26.321999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.49299999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.1239999999999997,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4861800448,"data_stored":9853775872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4700524544,"data_stored":9552097299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9566150656,"data_stored":19409698835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:05:56.953 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:05:56.954 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:06:09.939 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:06:09.933+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138090a90 2023-10-04T02:06:09.940 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-04T02:06:09.933+0000 7fd4d9ca6640 -1 reset not still connected to 0x55e138abdad0 2023-10-04T02:06:16.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:06:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:06:17.501 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:06:17.534 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2954,"stamp":"2023-10-04T02:06:15.927582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29417280057,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2556577,"num_read_kb":9430261,"num_write":2398147,"num_write_kb":9426127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102375,"ondisk_log_size":102375,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23171312,"kb_used_data":18742548,"kb_used_omap":5652,"kb_used_meta":4423083,"kb_avail":259944208,"statfs":{"total":289910292480,"available":266182868992,"internally_reserved":0,"allocated":19192369152,"data_stored":38874770088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4529237506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,5,6,16,17,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15966208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3808,"num_read_kb":15212,"num_write":3902,"num_write_kb":15608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001630"},"pg_stats":[{"pgid":"2.7","version":"164'9529","reported_seq":13346,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:11.579758+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:06:11.579758+0000","last_peered":"2023-10-04T02:06:11.579758+0000","last_clean":"2023-10-04T02:06:11.579758+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:06:11.579758+0000","last_undegraded":"2023-10-04T02:06:11.579758+0000","last_fullsized":"2023-10-04T02:06:11.579758+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"164'8025","reported_seq":11947,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:08.890508+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:06:08.890508+0000","last_peered":"2023-10-04T02:06:08.890508+0000","last_clean":"2023-10-04T02:06:08.890508+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:06:08.890508+0000","last_undegraded":"2023-10-04T02:06:08.890508+0000","last_fullsized":"2023-10-04T02:06:08.890508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"164'9684","reported_seq":13811,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:11.579530+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:06:11.579530+0000","last_peered":"2023-10-04T02:06:11.579530+0000","last_clean":"2023-10-04T02:06:11.579530+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:06:11.579530+0000","last_undegraded":"2023-10-04T02:06:11.579530+0000","last_fullsized":"2023-10-04T02:06:11.579530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"164'8113","reported_seq":13086,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:11.575267+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:06:11.575267+0000","last_peered":"2023-10-04T02:06:11.575267+0000","last_clean":"2023-10-04T02:06:11.575267+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:06:11.575267+0000","last_undegraded":"2023-10-04T02:06:11.575267+0000","last_fullsized":"2023-10-04T02:06:11.575267+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8113,"log_dups_size":0,"ondisk_log_size":8113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"164'600920","reported_seq":1185923,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:14.924820+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:06:14.924820+0000","last_peered":"2023-10-04T02:06:14.924820+0000","last_clean":"2023-10-04T02:06:14.924820+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:06:14.924820+0000","last_undegraded":"2023-10-04T02:06:14.924820+0000","last_fullsized":"2023-10-04T02:06:14.924820+0000","mapping_epoch":15,"log_start":"162'590873","ondisk_log_start":"162'590873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7389884416,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":589891,"num_read_kb":2327104,"num_write":589371,"num_write_kb":2357484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"164'19020","reported_seq":24838,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:14.747476+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:06:14.747476+0000","last_peered":"2023-10-04T02:06:14.747476+0000","last_clean":"2023-10-04T02:06:14.747476+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:06:14.747476+0000","last_undegraded":"2023-10-04T02:06:14.747476+0000","last_fullsized":"2023-10-04T02:06:14.747476+0000","mapping_epoch":12,"log_start":"101'9000","ondisk_log_start":"101'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"164'587090","reported_seq":1158519,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:12.057006+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:06:12.057006+0000","last_peered":"2023-10-04T02:06:12.057006+0000","last_clean":"2023-10-04T02:06:12.057006+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:06:12.057006+0000","last_undegraded":"2023-10-04T02:06:12.057006+0000","last_fullsized":"2023-10-04T02:06:12.057006+0000","mapping_epoch":15,"log_start":"162'577067","ondisk_log_start":"162'577067","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7232143360,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":576235,"num_read_kb":2273268,"num_write":575710,"num_write_kb":2302840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9897,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:08.779838+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:06:08.779838+0000","last_peered":"2023-10-04T02:06:08.779838+0000","last_clean":"2023-10-04T02:06:08.779838+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:06:08.779838+0000","last_undegraded":"2023-10-04T02:06:08.779838+0000","last_fullsized":"2023-10-04T02:06:08.779838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"164'613426","reported_seq":1211494,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:14.910608+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:06:14.910608+0000","last_peered":"2023-10-04T02:06:14.910608+0000","last_clean":"2023-10-04T02:06:14.910608+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:06:14.910608+0000","last_undegraded":"2023-10-04T02:06:14.910608+0000","last_fullsized":"2023-10-04T02:06:14.910608+0000","mapping_epoch":15,"log_start":"162'603366","ondisk_log_start":"162'603366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7565979648,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":603065,"num_read_kb":2379968,"num_write":601565,"num_write_kb":2406260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10232,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:08.807571+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:06:08.807571+0000","last_peered":"2023-10-04T02:06:08.807571+0000","last_clean":"2023-10-04T02:06:08.807571+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:06:08.807571+0000","last_undegraded":"2023-10-04T02:06:08.807571+0000","last_fullsized":"2023-10-04T02:06:08.807571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"164'586905","reported_seq":1158562,"reported_epoch":164,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:06:12.067984+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:06:12.067984+0000","last_peered":"2023-10-04T02:06:12.067984+0000","last_clean":"2023-10-04T02:06:12.067984+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:06:12.067984+0000","last_undegraded":"2023-10-04T02:06:12.067984+0000","last_fullsized":"2023-10-04T02:06:12.067984+0000","mapping_epoch":15,"log_start":"162'576863","ondisk_log_start":"162'576863","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5600,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 796s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7228665875,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":576212,"num_read_kb":2273068,"num_write":575857,"num_write_kb":2303422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"164'19888","reported_seq":25930,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:11.586483+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:06:11.586483+0000","last_peered":"2023-10-04T02:06:11.586483+0000","last_clean":"2023-10-04T02:06:11.586483+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:06:11.586483+0000","last_undegraded":"2023-10-04T02:06:11.586483+0000","last_fullsized":"2023-10-04T02:06:11.586483+0000","mapping_epoch":12,"log_start":"103'9800","ondisk_log_start":"103'9800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":348,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:46.960433+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:05:46.960433+0000","last_peered":"2023-10-04T02:05:46.960433+0000","last_clean":"2023-10-04T02:05:46.960433+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:05:46.960433+0000","last_undegraded":"2023-10-04T02:05:46.960433+0000","last_fullsized":"2023-10-04T02:05:46.960433+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29416673299,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2345403,"num_read_kb":9253408,"num_write":2342503,"num_write_kb":9370006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19185672192,"data_stored":38872768550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40172,"ondisk_log_size":40172,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":62104,"ondisk_log_size":62104,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739486,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11300204,"kb_used_data":9367520,"kb_used_omap":2858,"kb_used_meta":1929813,"kb_avail":83071636,"statfs":{"total":96636764160,"available":85065355264,"internally_reserved":0,"allocated":9592340480,"data_stored":19434540731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":1976129147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:14 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.44900000000000001,"15min":0.45400000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.53000000000000003,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.252},{"interface":"front","average":{"1min":0.373,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.20899999999999999},"max":{"1min":0.57499999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.316}]},{"osd":1,"last update":"Wed Oct 4 02:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.48099999999999998,"15min":0.53100000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1849999999999996,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.47999999999999998,"15min":0.57199999999999995},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":3.8370000000000002,"15min":35.423999999999999},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652024,"kb_used_data":4609008,"kb_used_omap":1357,"kb_used_meta":1041650,"kb_avail":88719816,"statfs":{"total":96636764160,"available":90849091584,"internally_reserved":0,"allocated":4719624192,"data_stored":9569322059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1066650552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,2,6,8,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32700000000000001,"15min":0.316},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.54100000000000004,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.49299999999999999,"15min":0.53100000000000003},"min":{"1min":0.218,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":0.629,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.309}]},{"osd":2,"last update":"Wed Oct 4 02:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.20999999999999999,"5min":0.192,"15min":0.17299999999999999},"max":{"1min":3.9900000000000002,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.59799999999999998,"15min":0.54500000000000004},"min":{"1min":0.246,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.7639999999999998,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739485,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6219084,"kb_used_data":4766020,"kb_used_omap":1437,"kb_used_meta":1451618,"kb_avail":88152756,"statfs":{"total":96636764160,"available":90268422144,"internally_reserved":0,"allocated":4880404480,"data_stored":9870907298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1486457807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,8,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46600000000000003,"15min":0.45100000000000001},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.26500000000000001,"5min":0.221,"15min":0.22},"max":{"1min":3.863,"5min":3.863,"15min":5.5110000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Oct 4 02:05:25 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53500000000000003,"15min":0.58199999999999996},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.219},"max":{"1min":1.9259999999999999,"5min":3.867,"15min":26.321999999999999},"last":0.44},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.49299999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.1239999999999997,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4878082048,"data_stored":9870057472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4716814336,"data_stored":9568387091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9590775808,"data_stored":19434323987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:06:17.535 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:06:17.536 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:06:37.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:06:38.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:06:38.014 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:06:38.047 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2965,"stamp":"2023-10-04T02:06:37.934541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29453820473,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2565519,"num_read_kb":9465937,"num_write":2407092,"num_write_kb":9461907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102462,"ondisk_log_size":102462,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23520188,"kb_used_data":18817508,"kb_used_omap":5652,"kb_used_meta":4697003,"kb_avail":259595332,"statfs":{"total":289910292480,"available":265825619968,"internally_reserved":0,"allocated":19269128192,"data_stored":38951529128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5788158,"internal_metadata":4809731586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,5,7,10,11,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20664320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5088,"num_read_kb":20304,"num_write":5060,"num_write_kb":20240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005325"},"pg_stats":[{"pgid":"2.7","version":"164'9559","reported_seq":13386,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:36.454394+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:06:36.454394+0000","last_peered":"2023-10-04T02:06:36.454394+0000","last_clean":"2023-10-04T02:06:36.454394+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:06:36.454394+0000","last_undegraded":"2023-10-04T02:06:36.454394+0000","last_fullsized":"2023-10-04T02:06:36.454394+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9559,"log_dups_size":0,"ondisk_log_size":9559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"164'8040","reported_seq":11967,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:35.610078+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:06:35.610078+0000","last_peered":"2023-10-04T02:06:35.610078+0000","last_clean":"2023-10-04T02:06:35.610078+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:06:35.610078+0000","last_undegraded":"2023-10-04T02:06:35.610078+0000","last_fullsized":"2023-10-04T02:06:35.610078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"164'9714","reported_seq":13851,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:36.450449+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:06:36.450449+0000","last_peered":"2023-10-04T02:06:36.450449+0000","last_clean":"2023-10-04T02:06:36.450449+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:06:36.450449+0000","last_undegraded":"2023-10-04T02:06:36.450449+0000","last_fullsized":"2023-10-04T02:06:36.450449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9714,"log_dups_size":0,"ondisk_log_size":9714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"164'8143","reported_seq":13126,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:36.449230+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:06:36.449230+0000","last_peered":"2023-10-04T02:06:36.449230+0000","last_clean":"2023-10-04T02:06:36.449230+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:06:36.449230+0000","last_undegraded":"2023-10-04T02:06:36.449230+0000","last_fullsized":"2023-10-04T02:06:36.449230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8143,"log_dups_size":0,"ondisk_log_size":8143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"164'602890","reported_seq":1189880,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:35.019895+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:06:35.019895+0000","last_peered":"2023-10-04T02:06:35.019895+0000","last_clean":"2023-10-04T02:06:35.019895+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:06:35.019895+0000","last_undegraded":"2023-10-04T02:06:35.019895+0000","last_fullsized":"2023-10-04T02:06:35.019895+0000","mapping_epoch":15,"log_start":"163'592873","ondisk_log_start":"163'592873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7397916672,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":591875,"num_read_kb":2335012,"num_write":591341,"num_write_kb":2365364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"164'19068","reported_seq":24898,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:34.379285+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:06:34.379285+0000","last_peered":"2023-10-04T02:06:34.379285+0000","last_clean":"2023-10-04T02:06:34.379285+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:06:34.379285+0000","last_undegraded":"2023-10-04T02:06:34.379285+0000","last_fullsized":"2023-10-04T02:06:34.379285+0000","mapping_epoch":12,"log_start":"101'9000","ondisk_log_start":"101'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"164'589615","reported_seq":1163564,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:37.275973+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:06:37.275973+0000","last_peered":"2023-10-04T02:06:37.275973+0000","last_clean":"2023-10-04T02:06:37.275973+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:06:37.275973+0000","last_undegraded":"2023-10-04T02:06:37.275973+0000","last_fullsized":"2023-10-04T02:06:37.275973+0000","mapping_epoch":15,"log_start":"163'579567","ondisk_log_start":"163'579567","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7242461184,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":578757,"num_read_kb":2283336,"num_write":578235,"num_write_kb":2312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9917,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:34.380435+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:06:34.380435+0000","last_peered":"2023-10-04T02:06:34.380435+0000","last_clean":"2023-10-04T02:06:34.380435+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:06:34.380435+0000","last_undegraded":"2023-10-04T02:06:34.380435+0000","last_fullsized":"2023-10-04T02:06:34.380435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"164'615413","reported_seq":1215484,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:35.014872+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:06:35.014872+0000","last_peered":"2023-10-04T02:06:35.014872+0000","last_clean":"2023-10-04T02:06:35.014872+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:06:35.014872+0000","last_undegraded":"2023-10-04T02:06:35.014872+0000","last_fullsized":"2023-10-04T02:06:35.014872+0000","mapping_epoch":15,"log_start":"163'605366","ondisk_log_start":"163'605366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7574081536,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":605067,"num_read_kb":2387964,"num_write":603552,"num_write_kb":2414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10247,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:34.378901+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:06:34.378901+0000","last_peered":"2023-10-04T02:06:34.378901+0000","last_clean":"2023-10-04T02:06:34.378901+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:06:34.378901+0000","last_undegraded":"2023-10-04T02:06:34.378901+0000","last_fullsized":"2023-10-04T02:06:34.378901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"164'589512","reported_seq":1163617,"reported_epoch":164,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:06:37.284517+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:06:37.284517+0000","last_peered":"2023-10-04T02:06:37.284517+0000","last_clean":"2023-10-04T02:06:37.284517+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:06:37.284517+0000","last_undegraded":"2023-10-04T02:06:37.284517+0000","last_fullsized":"2023-10-04T02:06:37.284517+0000","mapping_epoch":15,"log_start":"163'579463","ondisk_log_start":"163'579463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5771,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 821s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7238754323,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":578646,"num_read_kb":2282772,"num_write":578320,"num_write_kb":2313274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"164'19933","reported_seq":25990,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:36.489832+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:06:36.489832+0000","last_peered":"2023-10-04T02:06:36.489832+0000","last_clean":"2023-10-04T02:06:36.489832+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:06:36.489832+0000","last_undegraded":"2023-10-04T02:06:36.489832+0000","last_fullsized":"2023-10-04T02:06:36.489832+0000","mapping_epoch":12,"log_start":"103'9900","ondisk_log_start":"103'9900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":348,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:46.960433+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:05:46.960433+0000","last_peered":"2023-10-04T02:05:46.960433+0000","last_clean":"2023-10-04T02:05:46.960433+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:05:46.960433+0000","last_undegraded":"2023-10-04T02:05:46.960433+0000","last_fullsized":"2023-10-04T02:05:46.960433+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29453213715,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2354345,"num_read_kb":9289084,"num_write":2351448,"num_write_kb":9405786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19262832640,"data_stored":38949928998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40161,"ondisk_log_size":40161,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":62202,"ondisk_log_size":62202,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739491,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11451980,"kb_used_data":9406852,"kb_used_omap":2858,"kb_used_meta":2042261,"kb_avail":82919860,"statfs":{"total":96636764160,"available":84909936640,"internally_reserved":0,"allocated":9632616448,"data_stored":19474816699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2926981,"internal_metadata":2091275899},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:16 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.41299999999999998,"15min":0.45000000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.49399999999999999,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Wed Oct 4 02:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.48099999999999998,"15min":0.53100000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1849999999999996,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.438},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.47999999999999998,"15min":0.57199999999999995},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":3.8370000000000002,"15min":35.423999999999999},"last":0.28699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739489,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5779248,"kb_used_data":4628904,"kb_used_omap":1357,"kb_used_meta":1148978,"kb_avail":88592592,"statfs":{"total":96636764160,"available":90718814208,"internally_reserved":0,"allocated":4739997696,"data_stored":9589695563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1176554424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,6,9,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.41099999999999998,"15min":0.34300000000000003},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":7.016,"5min":7.016,"15min":7.016},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.57599999999999996,"15min":0.54700000000000004},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":7.0750000000000002,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17299999999999999},"max":{"1min":2.6419999999999999,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.67400000000000004,"15min":0.58799999999999997},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.6769999999999996,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":1.831}]}]},{"osd":0,"up_from":8,"seq":34359739489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6288960,"kb_used_data":4781752,"kb_used_omap":1437,"kb_used_meta":1505762,"kb_avail":88082880,"statfs":{"total":96636764160,"available":90196869120,"internally_reserved":0,"allocated":4896514048,"data_stored":9887016866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1541901263},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,3,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46600000000000003,"15min":0.45100000000000001},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.26500000000000001,"5min":0.221,"15min":0.22},"max":{"1min":3.863,"5min":3.863,"15min":5.5110000000000001},"last":0.223}]},{"osd":2,"last update":"Wed Oct 4 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.501,"15min":0.58199999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.219},"max":{"1min":1.8180000000000001,"5min":3.5019999999999998,"15min":26.321999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.59999999999999998,"15min":0.496},"min":{"1min":0.218,"5min":0.19,"15min":0.19},"max":{"1min":0.67800000000000005,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4894289920,"data_stored":9886265344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4737245184,"data_stored":9588817939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9631297536,"data_stored":19474845715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:06:38.049 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:06:38.049 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:06:38.883 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:38.876+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:06:38.892 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:38.884+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:06:38.892 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:38.884+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:06:38.892 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:38.884+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:06:38.895 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:38.888+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:06:40.765 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:40.752+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:06:40.774 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:06:40.756+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:06:58.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:06:58.490 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:06:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:06:58.523 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2975,"stamp":"2023-10-04T02:06:57.937966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29485814329,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2573480,"num_read_kb":9497661,"num_write":2414912,"num_write_kb":9493187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102774,"ondisk_log_size":102774,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23731112,"kb_used_data":18880080,"kb_used_omap":5654,"kb_used_meta":4845353,"kb_avail":259384408,"statfs":{"total":289910292480,"available":265609633792,"internally_reserved":0,"allocated":19333201920,"data_stored":39015602856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5789838,"internal_metadata":4961642354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,6,10,10,15,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20164608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5078,"num_read_kb":20232,"num_write":4926,"num_write_kb":19704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002051"},"pg_stats":[{"pgid":"2.7","version":"164'9590","reported_seq":13425,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:56.833280+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:06:56.833280+0000","last_peered":"2023-10-04T02:06:56.833280+0000","last_clean":"2023-10-04T02:06:56.833280+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:06:56.833280+0000","last_undegraded":"2023-10-04T02:06:56.833280+0000","last_fullsized":"2023-10-04T02:06:56.833280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9590,"log_dups_size":0,"ondisk_log_size":9590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"164'8057","reported_seq":11989,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:56.612336+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:06:56.612336+0000","last_peered":"2023-10-04T02:06:56.612336+0000","last_clean":"2023-10-04T02:06:56.612336+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:06:56.612336+0000","last_undegraded":"2023-10-04T02:06:56.612336+0000","last_fullsized":"2023-10-04T02:06:56.612336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"164'9745","reported_seq":13890,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:56.833570+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:06:56.833570+0000","last_peered":"2023-10-04T02:06:56.833570+0000","last_clean":"2023-10-04T02:06:56.833570+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:06:56.833570+0000","last_undegraded":"2023-10-04T02:06:56.833570+0000","last_fullsized":"2023-10-04T02:06:56.833570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"164'8175","reported_seq":13166,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:56.836852+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:06:56.836852+0000","last_peered":"2023-10-04T02:06:56.836852+0000","last_clean":"2023-10-04T02:06:56.836852+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:06:56.836852+0000","last_undegraded":"2023-10-04T02:06:56.836852+0000","last_fullsized":"2023-10-04T02:06:56.836852+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"164'604865","reported_seq":1193892,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:55.118692+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:06:55.118692+0000","last_peered":"2023-10-04T02:06:55.118692+0000","last_clean":"2023-10-04T02:06:55.118692+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:06:55.118692+0000","last_undegraded":"2023-10-04T02:06:55.118692+0000","last_fullsized":"2023-10-04T02:06:55.118692+0000","mapping_epoch":15,"log_start":"163'594773","ondisk_log_start":"163'594773","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7405989888,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":593913,"num_read_kb":2343132,"num_write":593316,"num_write_kb":2373264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"164'19095","reported_seq":24937,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:54.858763+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:06:54.858763+0000","last_peered":"2023-10-04T02:06:54.858763+0000","last_clean":"2023-10-04T02:06:54.858763+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:06:54.858763+0000","last_undegraded":"2023-10-04T02:06:54.858763+0000","last_fullsized":"2023-10-04T02:06:54.858763+0000","mapping_epoch":12,"log_start":"101'9000","ondisk_log_start":"101'9000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"164'591555","reported_seq":1167424,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:57.369493+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:06:57.369493+0000","last_peered":"2023-10-04T02:06:57.369493+0000","last_clean":"2023-10-04T02:06:57.369493+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:06:57.369493+0000","last_undegraded":"2023-10-04T02:06:57.369493+0000","last_fullsized":"2023-10-04T02:06:57.369493+0000","mapping_epoch":15,"log_start":"163'581467","ondisk_log_start":"163'581467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7250374656,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":580674,"num_read_kb":2290968,"num_write":580175,"num_write_kb":2320700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9933,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:54.870352+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:06:54.870352+0000","last_peered":"2023-10-04T02:06:54.870352+0000","last_clean":"2023-10-04T02:06:54.870352+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:06:54.870352+0000","last_undegraded":"2023-10-04T02:06:54.870352+0000","last_fullsized":"2023-10-04T02:06:54.870352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"164'617414","reported_seq":1219505,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:55.102632+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:06:55.102632+0000","last_peered":"2023-10-04T02:06:55.102632+0000","last_clean":"2023-10-04T02:06:55.102632+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:06:55.102632+0000","last_undegraded":"2023-10-04T02:06:55.102632+0000","last_fullsized":"2023-10-04T02:06:55.102632+0000","mapping_epoch":15,"log_start":"163'607366","ondisk_log_start":"163'607366","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7582273536,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":607087,"num_read_kb":2396024,"num_write":605553,"num_write_kb":2422212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10259,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:54.859323+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:06:54.859323+0000","last_peered":"2023-10-04T02:06:54.859323+0000","last_clean":"2023-10-04T02:06:54.859323+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:06:54.859323+0000","last_undegraded":"2023-10-04T02:06:54.859323+0000","last_fullsized":"2023-10-04T02:06:54.859323+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"164'591523","reported_seq":1167628,"reported_epoch":164,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-04T02:06:57.352322+0000","last_change":"2023-10-04T01:52:55.409549+0000","last_active":"2023-10-04T02:06:57.352322+0000","last_peered":"2023-10-04T02:06:57.352322+0000","last_clean":"2023-10-04T02:06:57.352322+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:06:57.352322+0000","last_undegraded":"2023-10-04T02:06:57.352322+0000","last_fullsized":"2023-10-04T02:06:57.352322+0000","mapping_epoch":15,"log_start":"163'581463","ondisk_log_start":"163'581463","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-04T00:33:23.877233+0000","last_clean_scrub_stamp":"2023-10-04T00:33:23.877233+0000","objects_scrubbed":5903,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 841s","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7246569491,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":580632,"num_read_kb":2290684,"num_write":580224,"num_write_kb":2320890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"164'19980","reported_seq":26049,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:56.863567+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:06:56.863567+0000","last_peered":"2023-10-04T02:06:56.863567+0000","last_clean":"2023-10-04T02:06:56.863567+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:06:56.863567+0000","last_undegraded":"2023-10-04T02:06:56.863567+0000","last_fullsized":"2023-10-04T02:06:56.863567+0000","mapping_epoch":12,"log_start":"103'9900","ondisk_log_start":"103'9900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":348,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-10-04T02:05:46.960433+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:05:46.960433+0000","last_peered":"2023-10-04T02:05:46.960433+0000","last_clean":"2023-10-04T02:05:46.960433+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:05:46.960433+0000","last_undegraded":"2023-10-04T02:05:46.960433+0000","last_fullsized":"2023-10-04T02:05:46.960433+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29485207571,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2362306,"num_read_kb":9320808,"num_write":2359268,"num_write_kb":9437066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19327049728,"data_stored":39014146086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40288,"ondisk_log_size":40288,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":62387,"ondisk_log_size":62387,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739495,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11535076,"kb_used_data":9438108,"kb_used_omap":2860,"kb_used_meta":2094099,"kb_avail":82836764,"statfs":{"total":96636764160,"available":84824846336,"internally_reserved":0,"allocated":9664622592,"data_stored":19506822843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2928661,"internal_metadata":2144358379},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:16 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.41299999999999998,"15min":0.45000000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.49399999999999999,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.191},{"interface":"front","average":{"1min":0.37,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.28199999999999997}]},{"osd":1,"last update":"Wed Oct 4 02:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.48099999999999998,"15min":0.53100000000000003},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1849999999999996,"5min":5.3319999999999999,"15min":33.975000000000001},"last":0.318},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.47999999999999998,"15min":0.57199999999999995},"min":{"1min":0.20399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.503,"5min":3.8370000000000002,"15min":35.423999999999999},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739493,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5848804,"kb_used_data":4644316,"kb_used_omap":1357,"kb_used_meta":1203122,"kb_avail":88523036,"statfs":{"total":96636764160,"available":90647588864,"internally_reserved":0,"allocated":4755779584,"data_stored":9605477451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1231997880},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,4,6,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.41099999999999998,"15min":0.34300000000000003},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":7.016,"5min":7.016,"15min":7.016},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.57599999999999996,"15min":0.54700000000000004},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":7.0750000000000002,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.246}]},{"osd":2,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17299999999999999},"max":{"1min":2.6419999999999999,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.67400000000000004,"15min":0.58799999999999997},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.6769999999999996,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739493,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6347232,"kb_used_data":4797656,"kb_used_omap":1437,"kb_used_meta":1548130,"kb_avail":88024608,"statfs":{"total":96636764160,"available":90137198592,"internally_reserved":0,"allocated":4912799744,"data_stored":9903302562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1585286095},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.46600000000000003,"15min":0.45100000000000001},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17100000000000001},"max":{"1min":0.505,"5min":5.1230000000000002,"15min":5.4850000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.26500000000000001,"5min":0.221,"15min":0.22},"max":{"1min":3.863,"5min":3.863,"15min":5.5110000000000001},"last":0.311}]},{"osd":2,"last update":"Wed Oct 4 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.501,"15min":0.58199999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.219},"max":{"1min":1.8180000000000001,"5min":3.5019999999999998,"15min":26.321999999999999},"last":0.38},{"interface":"front","average":{"1min":0.379,"5min":0.59999999999999998,"15min":0.496},"min":{"1min":0.218,"5min":0.19,"15min":0.19},"max":{"1min":0.67800000000000005,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4910583808,"data_stored":9902559232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4753006592,"data_stored":9604579347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9663459328,"data_stored":19507007507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:06:58.524 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-04T00:33:23.877233+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=0, tm_min=33, tm_sec=23, tm_wday=2, tm_yday=277, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=4, tm_hour=1, tm_min=38, tm_sec=49, tm_wday=2, tm_yday=277, tm_isdst=0) 2023-10-04T02:06:58.524 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-04T02:07:12.801 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.787+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251110 2023-10-04T02:07:12.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251790 2023-10-04T02:07:12.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c251ee0 2023-10-04T02:07:12.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c2a05b0 2023-10-04T02:07:12.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c5c6000 2023-10-04T02:07:12.803 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63c6545b0 2023-10-04T02:07:12.804 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.795+0000 7f29e174e640 -1 reset not still connected to 0x55a63cdd11e0 2023-10-04T02:07:12.807 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-04T02:07:12.799+0000 7f29e174e640 -1 reset not still connected to 0x55a63d3fedd0 2023-10-04T02:07:18.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-04T02:07:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-04T02:07:19.071 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-04T02:07:19.122 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":2986,"stamp":"2023-10-04T02:07:17.941415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29519254073,"num_objects":24562,"num_object_clones":19186,"num_object_copies":49124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24562,"num_whiteouts":0,"num_read":2581391,"num_read_kb":9529221,"num_write":2423067,"num_write_kb":9525807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102679,"ondisk_log_size":102679,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23398000,"kb_used_data":18945236,"kb_used_omap":5654,"kb_used_meta":4447081,"kb_avail":259717520,"statfs":{"total":289910292480,"available":265950740480,"internally_reserved":0,"allocated":19399921664,"data_stored":39082309139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5789838,"internal_metadata":4553811826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,3,3,10,4,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17272832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4105,"num_read_kb":16356,"num_write":4210,"num_write_kb":16840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001669"},"pg_stats":[{"pgid":"2.7","version":"165'9618","reported_seq":13463,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.812975+0000","last_change":"2023-10-04T01:39:01.533088+0000","last_active":"2023-10-04T02:07:15.812975+0000","last_peered":"2023-10-04T02:07:15.812975+0000","last_clean":"2023-10-04T02:07:15.812975+0000","last_became_active":"2023-10-04T00:33:18.862387+0000","last_became_peered":"2023-10-04T00:33:18.862387+0000","last_unstale":"2023-10-04T02:07:15.812975+0000","last_undegraded":"2023-10-04T02:07:15.812975+0000","last_fullsized":"2023-10-04T02:07:15.812975+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7505","last_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_deep_scrub":"141'7505","last_deep_scrub_stamp":"2023-10-04T01:39:01.532865+0000","last_clean_scrub_stamp":"2023-10-04T01:39:01.532865+0000","objects_scrubbed":26,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T07:40:36.291665+0000","scrub_duration":0.18717399300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13633,"num_read_kb":11385,"num_write":3196,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"165'8071","reported_seq":12008,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.339280+0000","last_change":"2023-10-04T01:38:56.643622+0000","last_active":"2023-10-04T02:07:15.339280+0000","last_peered":"2023-10-04T02:07:15.339280+0000","last_clean":"2023-10-04T02:07:15.339280+0000","last_became_active":"2023-10-04T00:33:18.861093+0000","last_became_peered":"2023-10-04T00:33:18.861093+0000","last_unstale":"2023-10-04T02:07:15.339280+0000","last_undegraded":"2023-10-04T02:07:15.339280+0000","last_fullsized":"2023-10-04T02:07:15.339280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'7000","last_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_deep_scrub":"140'7000","last_deep_scrub_stamp":"2023-10-04T01:38:56.643417+0000","last_clean_scrub_stamp":"2023-10-04T01:38:56.643417+0000","objects_scrubbed":18,"log_size":8071,"log_dups_size":0,"ondisk_log_size":8071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:11:51.850474+0000","scrub_duration":0.16825438700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":28735,"num_read_kb":24436,"num_write":7894,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2581,"num_omap_keys":57,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"165'9773","reported_seq":13927,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.802829+0000","last_change":"2023-10-04T01:39:04.485273+0000","last_active":"2023-10-04T02:07:15.802829+0000","last_peered":"2023-10-04T02:07:15.802829+0000","last_clean":"2023-10-04T02:07:15.802829+0000","last_became_active":"2023-10-04T00:33:18.862331+0000","last_became_peered":"2023-10-04T00:33:18.862331+0000","last_unstale":"2023-10-04T02:07:15.802829+0000","last_undegraded":"2023-10-04T02:07:15.802829+0000","last_fullsized":"2023-10-04T02:07:15.802829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7646","last_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_deep_scrub":"141'7646","last_deep_scrub_stamp":"2023-10-04T01:39:04.485090+0000","last_clean_scrub_stamp":"2023-10-04T01:39:04.485090+0000","objects_scrubbed":32,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:04.917005+0000","scrub_duration":0.205803614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2173,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":13985,"num_read_kb":11661,"num_write":3250,"num_write_kb":3132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2338,"num_omap_keys":28,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'8203","reported_seq":13203,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.800367+0000","last_change":"2023-10-04T01:38:59.568483+0000","last_active":"2023-10-04T02:07:15.800367+0000","last_peered":"2023-10-04T02:07:15.800367+0000","last_clean":"2023-10-04T02:07:15.800367+0000","last_became_active":"2023-10-04T00:33:18.861079+0000","last_became_peered":"2023-10-04T00:33:18.861079+0000","last_unstale":"2023-10-04T02:07:15.800367+0000","last_undegraded":"2023-10-04T02:07:15.800367+0000","last_fullsized":"2023-10-04T02:07:15.800367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'6077","last_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_deep_scrub":"141'6077","last_deep_scrub_stamp":"2023-10-04T01:38:59.568239+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.568239+0000","objects_scrubbed":36,"log_size":8203,"log_dups_size":0,"ondisk_log_size":8203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:48:16.057490+0000","scrub_duration":0.22101768899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2111,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5623,"num_read_kb":4827,"num_write":175,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2869,"num_omap_keys":68,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"165'606925","reported_seq":1197920,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.236572+0000","last_change":"2023-10-04T01:53:23.994766+0000","last_active":"2023-10-04T02:07:15.236572+0000","last_peered":"2023-10-04T02:07:15.236572+0000","last_clean":"2023-10-04T02:07:15.236572+0000","last_became_active":"2023-10-04T00:33:25.467237+0000","last_became_peered":"2023-10-04T00:33:25.467237+0000","last_unstale":"2023-10-04T02:07:15.236572+0000","last_undegraded":"2023-10-04T02:07:15.236572+0000","last_fullsized":"2023-10-04T02:07:15.236572+0000","mapping_epoch":15,"log_start":"163'596873","ondisk_log_start":"163'596873","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'524687","last_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_deep_scrub":"153'524687","last_deep_scrub_stamp":"2023-10-04T01:53:23.994539+0000","last_clean_scrub_stamp":"2023-10-04T01:53:23.994539+0000","objects_scrubbed":6077,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":860,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:09:52.320002+0000","scrub_duration":859.77537223800005,"objects_trimmed":35,"snaptrim_duration":3.4787173889999998,"stat_sum":{"num_bytes":7414448128,"num_objects":6077,"num_object_clones":4791,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":595881,"num_read_kb":2350996,"num_write":595376,"num_write_kb":2381504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"165'19137","reported_seq":24989,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:14.196842+0000","last_change":"2023-10-04T01:38:54.772180+0000","last_active":"2023-10-04T02:07:14.196842+0000","last_peered":"2023-10-04T02:07:14.196842+0000","last_clean":"2023-10-04T02:07:14.196842+0000","last_became_active":"2023-10-04T00:33:18.866107+0000","last_became_peered":"2023-10-04T00:33:18.866107+0000","last_unstale":"2023-10-04T02:07:14.196842+0000","last_undegraded":"2023-10-04T02:07:14.196842+0000","last_fullsized":"2023-10-04T02:07:14.196842+0000","mapping_epoch":12,"log_start":"103'9100","ondisk_log_start":"103'9100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'15901","last_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_deep_scrub":"140'15901","last_deep_scrub_stamp":"2023-10-04T01:38:54.771995+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.771995+0000","objects_scrubbed":31,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:16:55.034103+0000","scrub_duration":0.57389389400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1972,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":46404,"num_read_kb":38751,"num_write":13578,"num_write_kb":13355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4216,"num_omap_keys":59,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"165'593567","reported_seq":1171368,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:17.458430+0000","last_change":"2023-10-04T01:52:51.242650+0000","last_active":"2023-10-04T02:07:17.458430+0000","last_peered":"2023-10-04T02:07:17.458430+0000","last_clean":"2023-10-04T02:07:17.458430+0000","last_became_active":"2023-10-04T00:33:25.466974+0000","last_became_peered":"2023-10-04T00:33:25.466974+0000","last_unstale":"2023-10-04T02:07:17.458430+0000","last_undegraded":"2023-10-04T02:07:17.458430+0000","last_fullsized":"2023-10-04T02:07:17.458430+0000","mapping_epoch":15,"log_start":"163'583467","ondisk_log_start":"163'583467","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'510724","last_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_deep_scrub":"152'510724","last_deep_scrub_stamp":"2023-10-04T01:52:51.242468+0000","last_clean_scrub_stamp":"2023-10-04T01:52:51.242468+0000","objects_scrubbed":5988,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":824,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:15:24.749433+0000","scrub_duration":823.93254245100002,"objects_trimmed":86,"snaptrim_duration":4.7531639200000004,"stat_sum":{"num_bytes":7258624000,"num_objects":5988,"num_object_clones":4729,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":582605,"num_read_kb":2298668,"num_write":582187,"num_write_kb":2328748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"161'2107","reported_seq":9951,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.342102+0000","last_change":"2023-10-04T01:38:54.222837+0000","last_active":"2023-10-04T02:07:15.342102+0000","last_peered":"2023-10-04T02:07:15.342102+0000","last_clean":"2023-10-04T02:07:15.342102+0000","last_became_active":"2023-10-04T00:33:18.865982+0000","last_became_peered":"2023-10-04T00:33:18.865982+0000","last_unstale":"2023-10-04T02:07:15.342102+0000","last_undegraded":"2023-10-04T02:07:15.342102+0000","last_fullsized":"2023-10-04T02:07:15.342102+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'2099","last_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_deep_scrub":"140'2099","last_deep_scrub_stamp":"2023-10-04T01:38:54.222659+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.222659+0000","objects_scrubbed":37,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:10.148671+0000","scrub_duration":0.26231053199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2345,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":18239,"num_read_kb":15270,"num_write":3332,"num_write_kb":3159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4459,"num_omap_keys":55,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"165'619506","reported_seq":1223627,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.228123+0000","last_change":"2023-10-04T01:55:04.050525+0000","last_active":"2023-10-04T02:07:15.228123+0000","last_peered":"2023-10-04T02:07:15.228123+0000","last_clean":"2023-10-04T02:07:15.228123+0000","last_became_active":"2023-10-04T00:33:25.467086+0000","last_became_peered":"2023-10-04T00:33:25.467086+0000","last_unstale":"2023-10-04T02:07:15.228123+0000","last_undegraded":"2023-10-04T02:07:15.228123+0000","last_fullsized":"2023-10-04T02:07:15.228123+0000","mapping_epoch":15,"log_start":"163'609466","ondisk_log_start":"163'609466","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'545775","last_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_deep_scrub":"154'545775","last_deep_scrub_stamp":"2023-10-04T01:55:04.050332+0000","last_clean_scrub_stamp":"2023-10-04T01:55:04.050332+0000","objects_scrubbed":6256,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":956,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:53:35.516562+0000","scrub_duration":955.91875594099997,"objects_trimmed":45,"snaptrim_duration":3.4218126400000002,"stat_sum":{"num_bytes":7590858752,"num_objects":6256,"num_object_clones":4939,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":609117,"num_read_kb":2404116,"num_write":607645,"num_write_kb":2430580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"161'4538","reported_seq":10272,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.355508+0000","last_change":"2023-10-04T01:38:59.449186+0000","last_active":"2023-10-04T02:07:15.355508+0000","last_peered":"2023-10-04T02:07:15.355508+0000","last_clean":"2023-10-04T02:07:15.355508+0000","last_became_active":"2023-10-04T00:33:18.866133+0000","last_became_peered":"2023-10-04T00:33:18.866133+0000","last_unstale":"2023-10-04T02:07:15.355508+0000","last_undegraded":"2023-10-04T02:07:15.355508+0000","last_fullsized":"2023-10-04T02:07:15.355508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4532","last_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_deep_scrub":"141'4532","last_deep_scrub_stamp":"2023-10-04T01:38:59.448759+0000","last_clean_scrub_stamp":"2023-10-04T01:38:59.448759+0000","objects_scrubbed":31,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T10:49:29.372355+0000","scrub_duration":0.55502420699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1975,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":31087,"num_read_kb":25953,"num_write":8410,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"165'593582","reported_seq":1171676,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:17.474464+0000","last_change":"2023-10-04T02:07:08.930035+0000","last_active":"2023-10-04T02:07:17.474464+0000","last_peered":"2023-10-04T02:07:17.474464+0000","last_clean":"2023-10-04T02:07:17.474464+0000","last_became_active":"2023-10-04T00:33:25.467066+0000","last_became_peered":"2023-10-04T00:33:25.467066+0000","last_unstale":"2023-10-04T02:07:17.474464+0000","last_undegraded":"2023-10-04T02:07:17.474464+0000","last_fullsized":"2023-10-04T02:07:17.474464+0000","mapping_epoch":15,"log_start":"163'583563","ondisk_log_start":"163'583563","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'592704","last_scrub_stamp":"2023-10-04T02:07:08.929717+0000","last_deep_scrub":"165'592704","last_deep_scrub_stamp":"2023-10-04T02:07:08.929717+0000","last_clean_scrub_stamp":"2023-10-04T02:07:08.929717+0000","objects_scrubbed":5986,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":854,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T12:57:20.166779+0000","scrub_duration":853.47026588899996,"objects_trimmed":6,"snaptrim_duration":0.87294438900000004,"stat_sum":{"num_bytes":7254716435,"num_objects":5986,"num_object_clones":4727,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":582614,"num_read_kb":2298588,"num_write":582215,"num_write_kb":2328854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"165'20022","reported_seq":26104,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-10-04T02:07:15.835502+0000","last_change":"2023-10-04T01:38:55.906556+0000","last_active":"2023-10-04T02:07:15.835502+0000","last_peered":"2023-10-04T02:07:15.835502+0000","last_clean":"2023-10-04T02:07:15.835502+0000","last_became_active":"2023-10-04T00:33:18.860014+0000","last_became_peered":"2023-10-04T00:33:18.860014+0000","last_unstale":"2023-10-04T02:07:15.835502+0000","last_undegraded":"2023-10-04T02:07:15.835502+0000","last_fullsized":"2023-10-04T02:07:15.835502+0000","mapping_epoch":12,"log_start":"105'10000","ondisk_log_start":"105'10000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'16844","last_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_deep_scrub":"140'16844","last_deep_scrub_stamp":"2023-10-04T01:38:55.906355+0000","last_clean_scrub_stamp":"2023-10-04T01:38:55.906355+0000","objects_scrubbed":42,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:52:41.136416+0000","scrub_duration":0.45300327400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":53362,"num_read_kb":44479,"num_write":15672,"num_write_kb":15375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3724,"num_omap_keys":43,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"140'99","reported_seq":349,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-10-04T02:06:08.648093+0000","last_change":"2023-10-04T01:38:54.471038+0000","last_active":"2023-10-04T02:06:08.648093+0000","last_peered":"2023-10-04T02:06:08.648093+0000","last_clean":"2023-10-04T02:06:08.648093+0000","last_became_active":"2023-10-04T00:33:14.255384+0000","last_became_peered":"2023-10-04T00:33:14.255384+0000","last_unstale":"2023-10-04T02:06:08.648093+0000","last_undegraded":"2023-10-04T02:06:08.648093+0000","last_fullsized":"2023-10-04T02:06:08.648093+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'99","last_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_deep_scrub":"140'99","last_deep_scrub_stamp":"2023-10-04T01:38:54.470527+0000","last_clean_scrub_stamp":"2023-10-04T01:38:54.470527+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:17:28.967880+0000","scrub_duration":0.049628740999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29518647315,"num_objects":24307,"num_object_clones":19186,"num_object_copies":48614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24307,"num_whiteouts":0,"num_read":2370217,"num_read_kb":9352368,"num_write":2367423,"num_write_kb":9469686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19393667072,"data_stored":39080763430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40211,"ondisk_log_size":40211,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":16390,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":211068,"num_read_kb":176762,"num_write":55507,"num_write_kb":54227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25598,"num_omap_keys":378,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1884160,"data_stored":32780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60436,"internal_metadata":0},"log_size":62369,"ondisk_log_size":62369,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739499,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11074996,"kb_used_data":9470696,"kb_used_omap":2860,"kb_used_meta":1601427,"kb_avail":83296844,"statfs":{"total":96636764160,"available":85295968256,"internally_reserved":0,"allocated":9697992704,"data_stored":19540188468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2928661,"internal_metadata":1639862251},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:16 2023","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.41299999999999998,"15min":0.45000000000000001},"min":{"1min":0.17000000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.49399999999999999,"5min":3.8170000000000002,"15min":6.7199999999999998},"last":0.38},{"interface":"front","average":{"1min":0.37,"5min":0.48199999999999998,"15min":0.50900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":3.8519999999999999,"15min":6.7930000000000001},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Oct 4 02:07:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.46100000000000002,"15min":0.44500000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.2949999999999999,"5min":5.1849999999999996,"15min":5.3319999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.97399999999999998,"5min":3.8370000000000002,"15min":6.4249999999999998},"last":0.22}]}]},{"osd":1,"up_from":8,"seq":34359739497,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5923076,"kb_used_data":4660284,"kb_used_omap":1357,"kb_used_meta":1261426,"kb_avail":88448764,"statfs":{"total":96636764160,"available":90571534336,"internally_reserved":0,"allocated":4772130816,"data_stored":9621824196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1389640,"internal_metadata":1291701176},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.41099999999999998,"15min":0.34300000000000003},"min":{"1min":0.24399999999999999,"5min":0.187,"15min":0.187},"max":{"1min":7.016,"5min":7.016,"15min":7.016},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.57599999999999996,"15min":0.54700000000000004},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.20300000000000001},"max":{"1min":7.0750000000000002,"5min":8.7409999999999997,"15min":8.7409999999999997},"last":0.441}]},{"osd":2,"last update":"Wed Oct 4 02:06:19 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17299999999999999},"max":{"1min":2.6419999999999999,"5min":8.5760000000000005,"15min":8.5760000000000005},"last":0.249},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.67400000000000004,"15min":0.58799999999999997},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.6769999999999996,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6399928,"kb_used_data":4814256,"kb_used_omap":1437,"kb_used_meta":1584226,"kb_avail":87971912,"statfs":{"total":96636764160,"available":90083237888,"internally_reserved":0,"allocated":4929798144,"data_stored":9920296475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1471537,"internal_metadata":1622248399},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,2,7,3,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 4 02:07:02 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":0.60599999999999998,"15min":0.499},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":14.432,"5min":14.432,"15min":14.432},"last":0.311},{"interface":"front","average":{"1min":1.117,"5min":0.64400000000000002,"15min":0.55600000000000005},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":14.695,"5min":14.695,"15min":14.695},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Oct 4 02:06:27 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.501,"15min":0.58199999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.219},"max":{"1min":1.8180000000000001,"5min":3.5019999999999998,"15min":26.321999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.379,"5min":0.59999999999999998,"15min":0.496},"min":{"1min":0.218,"5min":0.19,"15min":0.19},"max":{"1min":0.67800000000000005,"5min":4.9770000000000003,"15min":5.2309999999999999},"last":0.63300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":9072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18337,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":16390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25980,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":7318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16119,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4927578112,"data_stored":9919553536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4769386496,"data_stored":9620959251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9696702464,"data_stored":19540250643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-04T02:07:19.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-04T02:07:19.710 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-04T02:07:19.710 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-04T02:07:19.710 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-04T02:07:19.710 INFO:teuthology.orchestra.run:waiting for 300 2023-10-04T02:07:25.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-04T02:07:25.813 INFO:tasks.ceph.osd.0:Stopped 2023-10-04T02:07:25.813 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-04T02:07:25.814 INFO:teuthology.orchestra.run:waiting for 300 2023-10-04T02:07:30.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-04T02:07:30.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-04T02:07:31.226 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T02:07:31.219+0000 7fe2d7cf4640 -1 reset not still connected to 0x563023a99860 2023-10-04T02:07:31.226 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T02:07:31.219+0000 7fe2d7cf4640 -1 reset not still connected to 0x5630245bbe10 2023-10-04T02:07:31.226 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-04T02:07:31.219+0000 7fe2d7cf4640 -1 reset not still connected to 0x56302e79eb60 2023-10-04T02:07:31.915 INFO:tasks.ceph.osd.1:Stopped 2023-10-04T02:07:31.916 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-04T02:07:31.916 INFO:teuthology.orchestra.run:waiting for 300 2023-10-04T02:07:36.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-04T02:07:36.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-04T02:07:38.017 INFO:tasks.ceph.osd.2:Stopped 2023-10-04T02:07:38.017 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-04T02:07:38.018 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-04T02:07:38.018 INFO:teuthology.orchestra.run:waiting for 300 2023-10-04T02:07:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-04T02:07:41.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-04T02:07:44.119 INFO:tasks.ceph.mgr.x:Stopped 2023-10-04T02:07:44.119 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-04T02:07:44.120 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-04T02:07:44.120 INFO:teuthology.orchestra.run:waiting for 300 2023-10-04T02:07:46.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-04T02:07:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-04T02:07:50.221 INFO:tasks.ceph.mon.a:Stopped 2023-10-04T02:07:50.222 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-04T02:07:50.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-04T02:07:50.821 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T02:07:50.822 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-04T02:07:51.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-04T02:07:52.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T02:07:52.383 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-04T02:07:52.735 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T02:07:52.735 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-04T02:07:53.193 INFO:tasks.ceph:Archiving mon data... 2023-10-04T02:07:53.195 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079/data/mon.a.tgz 2023-10-04T02:07:53.195 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-10-04T02:07:53.219 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.b2iXMSqD9H 2023-10-04T02:07:53.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.b2iXMSqD9H 2023-10-04T02:07:53.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.b2iXMSqD9H 2023-10-04T02:07:53.485 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.b2iXMSqD9H is 580KB 2023-10-04T02:07:53.550 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.b2iXMSqD9H 2023-10-04T02:07:53.556 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-04T02:07:53.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-04T02:07:53.637 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-04T02:07:53.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-04T02:07:53.709 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-04T02:07:53.714 INFO:tasks.ceph:Archiving crash dumps... 2023-10-04T02:07:53.716 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079/remote/smithi027/crash 2023-10-04T02:07:53.717 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-04T02:07:53.766 INFO:tasks.ceph:Compressing logs... 2023-10-04T02:07:53.767 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-04T02:07:53.834 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41989.log 2023-10-04T02:07:53.834 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42928.log 2023-10-04T02:07:53.835 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44622.log 2023-10-04T02:07:53.835 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43268.log 2023-10-04T02:07:53.835 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55988.log 2023-10-04T02:07:53.836 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42928.log: /var/log/ceph/ceph-client.admin.43268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45633.log 2023-10-04T02:07:53.836 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55988.log: /var/log/ceph/ceph-client.admin.45633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55988.log.gz 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43141.log 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47383.log 2023-10-04T02:07:53.838 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2023-10-04T02:07:53.841 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55091.log 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55723.log 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55356.log 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55091.log.gz 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54214.log 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55723.log.gz 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42972.log 2023-10-04T02:07:53.842 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55356.log.gz 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52589.log 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54214.log.gz 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42972.log: /var/log/ceph/ceph-client.admin.52589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52589.log.gz 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46294.log 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2023-10-04T02:07:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2023-10-04T02:07:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56227.log 2023-10-04T02:07:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55541.log 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40721.log 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56227.log.gz 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55541.log.gz 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45573.log 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2023-10-04T02:07:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50064.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45573.log: /var/log/ceph/ceph-client.admin.41737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54874.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54874.log.gz 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47760.log 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2023-10-04T02:07:53.850 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46492.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53978.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.50215.log: --verbose -- /var/log/ceph/ceph-client.admin.54292.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50701.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53978.log.gz 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55486.log 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54292.log.gz 2023-10-04T02:07:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50701.log: /var/log/ceph/ceph-client.admin.55486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57322.log 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55486.log.gz 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52802.log 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49941.log 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57322.log.gz 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52802.log.gz 2023-10-04T02:07:53.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47023.log 2023-10-04T02:07:53.863 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34182.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42098.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34182.log.gz 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53924.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38922.log 2023-10-04T02:07:53.864 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53924.log.gz 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45603.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44127.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38922.log.gz 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47353.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40397.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47353.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50731.log 2023-10-04T02:07:53.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42260.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48114.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42232.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53525.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53525.log.gz 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47788.log 2023-10-04T02:07:53.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46662.log 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57213.log 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40397.log.gz 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41989.log.gz 2023-10-04T02:07:53.867 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46418.log 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42098.log.gz 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44622.log.gz 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42260.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46294.log.gz 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44127.log.gz 2023-10-04T02:07:53.868 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43268.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42232.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57213.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43141.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46492.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47760.log.gz 2023-10-04T02:07:53.869 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57213.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42972.log.gz 0.0% 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45633.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49941.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45573.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45603.log.gz 2023-10-04T02:07:53.870 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50638.log 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48114.log.gz 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50701.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46662.log.gz 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47023.log.gz 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47353.log.gz 0.0% 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47383.log.gz 2023-10-04T02:07:53.871 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50064.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51979.log 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47788.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50638.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57269.log 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51979.log.gz 2023-10-04T02:07:53.872 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50668.log 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr: 26.7%/var/log/ceph/ceph-client.admin.57269.log: -- replaced with /var/log/ceph/ceph-client.admin.50731.log.gz 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56308.log 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57269.log.gz 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50668.log.gz 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.56308.log: /var/log/ceph/ceph-client.admin.41861.log 2023-10-04T02:07:53.873 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56308.log.gz 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54050.log 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43768.log 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42432.log 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54050.log.gz 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44237.log 2023-10-04T02:07:53.874 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43768.log.gz 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53797.log 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42432.log.gz 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51129.log 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44237.log.gz 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42041.log 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53797.log.gz 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50548.log 2023-10-04T02:07:53.875 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51129.log.gz 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42041.log.gz 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56736.log 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50548.log.gz 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51491.log 2023-10-04T02:07:53.876 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56736.log.gz 2023-10-04T02:07:53.877 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2023-10-04T02:07:53.877 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2023-10-04T02:07:53.877 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43489.log 2023-10-04T02:07:53.877 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51491.log.gz 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43356.log 2023-10-04T02:07:53.933 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43489.log.gz 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52434.log 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34300.log 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43356.log.gz 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54795.log 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52434.log.gz 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2023-10-04T02:07:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54795.log.gz 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2023-10-04T02:07:53.935 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53344.log 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42017.log: /var/log/ceph/ceph-client.admin.54162.log 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53344.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53344.log.gz -- 2023-10-04T02:07:53.936 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.53223.log 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46073.log 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54162.log.gz 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45248.log 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53223.log.gz 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51949.log 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46073.log.gz 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45248.log.gz 2023-10-04T02:07:53.937 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51949.log.gz 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52740.log 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46168.log 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53687.log 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52740.log.gz 2023-10-04T02:07:53.938 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51858.log 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46168.log.gz 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45546.log 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53687.log.gz 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43325.log 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51858.log.gz 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55006.log 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45546.log.gz 2023-10-04T02:07:53.939 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56201.log 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43325.log.gz 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45994.log 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55006.log.gz 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55858.log 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56201.log.gz 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42188.log 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45994.log.gz 2023-10-04T02:07:53.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46328.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55858.log.gz 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55117.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44483.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48681.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51008.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56627.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2023-10-04T02:07:53.941 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40201.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39857.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46819.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56416.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46767.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51888.log 2023-10-04T02:07:53.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49691.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49789.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43985.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44838.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56468.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55671.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56892.log 2023-10-04T02:07:53.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45003.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41833.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40309.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48570.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45663.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54822.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2023-10-04T02:07:53.944 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53043.log 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42188.log: /var/log/ceph/ceph-client.admin.46328.log: /var/log/ceph/ceph-client.admin.55117.log: /var/log/ceph/ceph-client.admin.43665.log: /var/log/ceph/ceph-client.admin.44483.log: /var/log/ceph/ceph-client.admin.48681.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42188.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46328.log.gz 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44483.log.gz 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48681.log.gz 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55117.log.gz 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43631.log 2023-10-04T02:07:53.945 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51008.log: /var/log/ceph/ceph-client.admin.53043.log: /var/log/ceph/ceph-client.admin.56627.log: /var/log/ceph/ceph-client.admin.40795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51008.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.2.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53043.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56627.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40201.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40073.log 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46819.log: /var/log/ceph/ceph-client.admin.56416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39857.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51612.log: /var/log/ceph/ceph-client.admin.46767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46819.log.gz 2023-10-04T02:07:53.946 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56416.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51888.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46767.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42158.log: /var/log/ceph/ceph-client.admin.49691.log: /var/log/ceph/ceph-client.admin.43631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51888.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47596.log 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49691.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43631.log.gz -- replaced with /var/log/ceph/ceph-client.admin.49789.log.gz 2023-10-04T02:07:53.947 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43985.log: /var/log/ceph/ceph-client.admin.44838.log: /var/log/ceph/ceph-client.admin.56468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43985.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44838.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56468.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55671.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56892.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45003.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40073.log: /var/log/ceph/ceph-client.admin.41833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40309.log: /var/log/ceph/ceph-client.admin.42341.log: /var/log/ceph/ceph-client.admin.48570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45003.log.gz 2023-10-04T02:07:53.948 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40073.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41833.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48570.log.gz 0.0% 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40309.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45663.log: /var/log/ceph/ceph-client.admin.54822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45663.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54822.log.gz 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47596.log.gz -5 2023-10-04T02:07:53.949 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47870.log 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40056.log 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46135.log 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47870.log.gz 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54769.log 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40056.log.gz 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34300.log.gz 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53284.log 2023-10-04T02:07:53.950 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46135.log.gz 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52953.log 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54769.log.gz 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55143.log 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53284.log.gz 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51674.log 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52953.log.gz 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56521.log 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55143.log.gz 2023-10-04T02:07:53.951 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51674.log.gz 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55382.log 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56521.log.gz 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48961.log 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44930.log 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55382.log.gz 2023-10-04T02:07:53.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48961.log.gz 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43171.log 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44930.log.gz 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43171.log.gz 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54370.log 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51429.log 2023-10-04T02:07:53.953 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47443.log 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54370.log.gz 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.51429.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51429.log.gz 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.47443.log: --verbose -- /var/log/ceph/ceph-client.admin.50796.log 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47443.log.gz 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55408.log 2023-10-04T02:07:53.954 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50796.log.gz 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54848.log 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55408.log.gz 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55224.log 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55460.log 2023-10-04T02:07:53.955 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54848.log.gz 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46877.log 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55224.log.gz 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55460.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.42683.log -- replaced with /var/log/ceph/ceph-client.admin.55460.log.gz 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49520.log 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46877.log.gz 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45800.log 2023-10-04T02:07:53.956 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42683.log.gz 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49520.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49520.log.gz 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51339.log 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47686.log 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45800.log.gz 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57134.log 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51339.log.gz 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47686.log.gz 2023-10-04T02:07:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46544.log 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57134.log.gz 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42838.log 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46544.log.gz 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53133.log 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42838.log.gz 2023-10-04T02:07:53.958 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54926.log 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53133.log.gz 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54926.log.gz 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2023-10-04T02:07:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53996.log 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49544.log 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42868.log 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53996.log.gz 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42401.log 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49544.log.gz 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55775.log 2023-10-04T02:07:53.960 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42868.log.gz 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42401.log.gz 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44067.log 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55775.log.gz 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53906.log 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45084.log 2023-10-04T02:07:53.961 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44067.log.gz 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49661.log 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53906.log.gz 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53314.log 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45084.log.gz 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49661.log.gz 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2023-10-04T02:07:53.962 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53314.log.gz 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46940.log 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56120.log 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53647.log 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46940.log.gz 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48033.log 2023-10-04T02:07:53.963 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56120.log.gz 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41915.log 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53647.log.gz 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48033.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.48033.log.gz 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.48759.log 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44157.log 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41915.log.gz 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48759.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30591.log 2023-10-04T02:07:53.964 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48759.log.gz 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46795.log 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44157.log.gz 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log/var/log/ceph/ceph.tmp-client.admin.30591.log: 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30591.log.gz 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46795.log.gz 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2023-10-04T02:07:53.965 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50185.log 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50185.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.55910.log 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50185.log.gz 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54637.log 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2023-10-04T02:07:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42757.log 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54637.log.gzgzip 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52192.log 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2023-10-04T02:07:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45114.log 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52192.log.gz 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51399.log 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45114.log.gz 2023-10-04T02:07:53.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56918.log 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51399.log.gz 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54110.log 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40147.log 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56918.log.gz 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54110.log: -- /var/log/ceph/ceph-client.admin.57053.log 2023-10-04T02:07:53.969 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54110.log.gz 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40147.log.gz 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57053.log.gz 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50608.log 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2023-10-04T02:07:53.970 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53619.log 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50608.log.gz 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49115.log 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53619.log.gz 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56654.log 2023-10-04T02:07:53.971 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51736.log 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49115.log.gz 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56654.log.gz 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52650.log 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43881.log 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55619.log 2023-10-04T02:07:53.972 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52650.log.gz 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48144.log 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43881.log.gz 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43000.log 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55619.log.gz 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53942.log 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48144.log.gz 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46520.log 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43000.log.gz 2023-10-04T02:07:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56601.log 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53942.log.gz 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44780.log 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46520.log.gz 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50978.log 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56601.log.gz 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45327.log 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44780.log: /var/log/ceph/ceph-client.admin.50978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41055.log 2023-10-04T02:07:53.974 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44780.log.gz 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50978.log.gz 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55962.log 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45327.log.gz 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41055.log.gz 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53588.log 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55962.log.gz 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43851.log 2023-10-04T02:07:53.975 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53588.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53588.log.gz -5 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56040.log 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53761.log 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43851.log.gz 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52620.log 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56040.log.gz 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40505.log 2023-10-04T02:07:53.976 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53761.log.gz 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46632.log 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52620.log.gz 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41959.log 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40505.log.gz 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46632.log.gz 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41959.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.41959.log.gz /var/log/ceph/ceph-client.admin.40434.log 2023-10-04T02:07:53.977 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45858.log 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39965.log 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45858.log.gz 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48332.log 2023-10-04T02:07:53.978 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39965.log.gz 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43820.log 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54980.log 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48332.log.gz 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.43820.log: --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43820.log.gz 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43024.log 2023-10-04T02:07:53.979 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54980.log.gz 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55830.log 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45494.log 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43024.log.gz 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44187.log 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55830.log.gz 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45494.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45494.log.gz 2023-10-04T02:07:53.980 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55276.log 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42529.log 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44187.log.gz 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46693.log 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55276.log.gz 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51581.log 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42529.log.gz 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56575.log 2023-10-04T02:07:53.981 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46693.log.gz 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43416.log 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51581.log.gz 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54716.log 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56575.log.gz 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56681.log 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43416.log.gz 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54716.log.gz 2023-10-04T02:07:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56681.log.gz 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2023-10-04T02:07:53.983 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gz 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42476.log 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50368.log 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42476.log.gz 2023-10-04T02:07:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52404.log 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55512.log 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50368.log.gz 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40181.log 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52404.log.gz 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49328.log 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55512.log.gz 2023-10-04T02:07:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49418.log 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40181.log.gz 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48176.log 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49328.log.gz 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51706.log 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49418.log.gz 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.48176.log: --verbose -- /var/log/ceph/ceph-client.admin.53723.log 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48176.log.gz 2023-10-04T02:07:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51706.log.gz 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53723.log.gz 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39948.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48651.log 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2023-10-04T02:07:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39985.log 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48651.log.gz 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44900.log 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39985.log.gz 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47110.log 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44900.log.gz 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2023-10-04T02:07:53.988 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2023-10-04T02:07:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52132.log 2023-10-04T02:07:53.989 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47110.log.gz 2023-10-04T02:07:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45522.log 2023-10-04T02:07:53.989 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2023-10-04T02:07:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52132.log.gz 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49175.log 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45522.log.gz 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38976.log 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2023-10-04T02:07:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52923.log 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49175.log.gz 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56282.log 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38976.log.gz 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43081.log 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52923.log.gz 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57241.log 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56282.log.gz 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49277.log 2023-10-04T02:07:54.075 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43081.log.gz 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57241.log.gz 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56495.log 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49277.log.gz 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2023-10-04T02:07:54.076 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56495.log.gz 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45144.log 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54266.log 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57346.log 2023-10-04T02:07:54.077 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45144.log.gz 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50125.log 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54266.log.gz 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57346.log.gz 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.50125.log: --verbose -- /var/log/ceph/ceph-client.admin.50428.log 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50125.log.gz 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.50488.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45776.log 2023-10-04T02:07:54.078 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38896.log 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50428.log.gz 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39891.log 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45776.log.gz 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38896.log.gz 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56066.log 2023-10-04T02:07:54.079 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39891.log.gz 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56066.log.gz 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51068.log 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50335.log 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2023-10-04T02:07:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50763.log 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51068.log.gz 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57105.log 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50335.log.gz 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50034.log 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50763.log.gz 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57105.log.gz 2023-10-04T02:07:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50034.log.gz 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47413.log 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47930.log 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52863.log 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47413.log.gz 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50245.log 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47930.log.gz 2023-10-04T02:07:54.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52863.log.gz 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43215.log 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50245.log.gz 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51249.log 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43215.log.gz 2023-10-04T02:07:54.083 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54585.log 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51249.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51249.log.gz 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51643.log 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54585.log.gz 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56334.log 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44755.log 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51643.log.gz 2023-10-04T02:07:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47626.log 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56334.log.gz 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44755.log.gz 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45300.log 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47626.log.gz 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2023-10-04T02:07:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52072.log 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45300.log.gz 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51309.log 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51219.log 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52072.log.gz 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51099.log 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51309.log.gz 2023-10-04T02:07:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51219.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51219.log.gz 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47473.log 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51099.log.gz 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56175.log 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43570.log 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47473.log.gz 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48711.log 2023-10-04T02:07:54.087 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56175.log.gz 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42781.log 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43570.log.gz 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54397.log 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48711.log.gz 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42781.log.gz 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43513.log 2023-10-04T02:07:54.088 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54397.log.gz 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52343.log 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47274.log 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43513.log.gz 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54507.log 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52343.log.gz 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39288.log 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2023-10-04T02:07:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54507.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54507.log.gz -5 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54453.log 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54014.log 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39288.log.gz 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.54453.log: --verbose -- /var/log/ceph/ceph-client.admin.49033.log 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54453.log.gz 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49249.log 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54014.log.gz 2023-10-04T02:07:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49033.log.gz 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49249.log.gz 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54690.log 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49877.log 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2023-10-04T02:07:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54690.log.gz 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49877.log.gz 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52102.log 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56549.log 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2023-10-04T02:07:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51521.log 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52102.log.gz 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57027.log 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56549.log.gz 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52833.log 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51521.log.gz 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44318.log 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57027.log.gz 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46358.log 2023-10-04T02:07:54.093 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52833.log.gz 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49085.log 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44318.log.gz 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr: 89.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54318.log 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46358.log.gz 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44562.log 2023-10-04T02:07:54.094 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49085.log.gz 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54136.log 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54318.log.gz 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44562.log.gz 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54136.log.gz 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55645.log 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2023-10-04T02:07:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51767.log 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55645.log.gz 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51767.log.gz 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2023-10-04T02:07:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53163.log 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44348.log 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53163.log.gz 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45276.log 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44348.log.gz 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56762.log 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2023-10-04T02:07:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43243.log 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45204.log 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56762.log.gz 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42621.log 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43243.log.gz 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50948.log 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45204.log.gz 2023-10-04T02:07:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54069.log 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42621.log.gz 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50948.log.gz 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54069.log.gz 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2023-10-04T02:07:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55593.log 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48206.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz -- 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.56442.log 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55593.log.gz 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53556.log 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56442.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.56442.log.gz /var/log/ceph/ceph-client.admin.50458.log 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47900.log 2023-10-04T02:07:54.100 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53556.log.gz 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50458.log.gz 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45948.log 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52893.log 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47900.log.gz 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45948.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45948.log.gz 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.43051.log 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2023-10-04T02:07:54.101 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52893.log.gz 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52282.log 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43051.log.gz 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53742.log 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39928.log 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48084.log 2023-10-04T02:07:54.102 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53742.log.gz 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39928.log.gz 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48084.log.gz 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56146.log 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55329.log 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2023-10-04T02:07:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44266.log 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56146.log.gz 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57296.log 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55329.log.gz 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44266.log.gz 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42371.log 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57296.log.gz 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43295.log 2023-10-04T02:07:54.104 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42371.log.gz 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40289.log 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43295.log.gz 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42652.log 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2023-10-04T02:07:54.105 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40289.log.gz 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48501.log 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48621.log 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52983.log 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48501.log.gz 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2023-10-04T02:07:54.106 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48621.log.gz 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52983.log.gz 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52557.log 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44378.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47569.log 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46198.log 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52557.log.gz 2023-10-04T02:07:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53870.log 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47569.log.gz 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53815.log 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46198.log.gz 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56708.log 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53870.log.gz 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41181.log 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53815.log.gz 2023-10-04T02:07:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48396.log 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56708.log.gz 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41181.log.gz 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48396.log.gz 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49005.log 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2023-10-04T02:07:54.109 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54423.log 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49005.log.gz 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54423.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54423.log.gz -5 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49492.log 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47517.log 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2023-10-04T02:07:54.110 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.49492.log: --verbose -- /var/log/ceph/ceph-client.admin.54743.log 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49492.log.gz 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47302.log 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47517.log.gz 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53465.log 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54743.log.gz 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49145.log 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47302.log.gz 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2023-10-04T02:07:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53465.log.gz 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54188.log 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49145.log.gz 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40866.log 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54188.log.gz 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53888.log 2023-10-04T02:07:54.112 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40866.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40866.log.gz 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47200.log 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53888.log.gz 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56840.log 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47056.log 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47200.log.gz 2023-10-04T02:07:54.113 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56840.log.gz 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56866.log 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47056.log.gz 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56866.log.gz 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50856.log 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2023-10-04T02:07:54.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41527.log 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51159.log 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43540.log 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41527.log.gz 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47961.log 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51159.log.gz 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48279.log 2023-10-04T02:07:54.115 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43540.log.gz 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54087.log 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47961.log.gz 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55697.log 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48279.log.gz 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50305.log 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54087.log.gz 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53254.log 2023-10-04T02:07:54.116 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55697.log.gz 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50305.log.gz 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41401.log 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53254.log.gz 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55884.log 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53193.log 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41401.log.gz 2023-10-04T02:07:54.117 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55936.log 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54344.log 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53193.log.gz 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.55936.log: -- /var/log/ceph/ceph-client.admin.56256.log 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55936.log.gz 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54344.log.gz 2023-10-04T02:07:54.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44808.log 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56256.log.gz 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54481.log 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50004.log 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44808.log.gz 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48542.log 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54481.log.gz 2023-10-04T02:07:54.119 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50095.log 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50004.log.gz 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49205.log 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48542.log.gz 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56092.log 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50095.log.gz 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44511.log 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49205.log.gz 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55567.log 2023-10-04T02:07:54.120 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56092.log.gz 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47839.log 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55567.log.gz 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55434.log 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47839.log.gz 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50578.log 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2023-10-04T02:07:54.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55434.log.gz 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45174.log 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50578.log.gz 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57001.log 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45174.log.gz 2023-10-04T02:07:54.122 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52710.log 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57001.log.gz 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46571.log 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52710.log.gz 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52771.log 2023-10-04T02:07:54.123 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46571.log.gz 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48594.log 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39314.log 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52771.log.gz 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46388.log 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48594.log.gz 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42729.log 2023-10-04T02:07:54.124 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39314.log.gz 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46388.log.gz 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45418.log 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42729.log.gz 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44868.log 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53073.log 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45418.log.gz 2023-10-04T02:07:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46847.log 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44868.log.gz 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43461.log 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53073.log.gz 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39002.log 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46847.log.gz 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48841.log 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43461.log.gz 2023-10-04T02:07:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55032.log 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39002.log.gz 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44975.log 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48841.log.gz 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55032.log.gz 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54900.log 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44975.log.gz 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2023-10-04T02:07:54.127 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54900.log.gz 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57160.log 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48251.log 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42808.log 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57160.log.gz 2023-10-04T02:07:54.128 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57186.log 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48251.log.gz 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42898.log 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42808.log.gz 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57186.log.gz 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54663.log 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44727.log 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42898.log.gz 2023-10-04T02:07:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46910.log 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54663.log.gz 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44727.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44727.log.gz -5 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47326.log 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46910.log.gz 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48789.log 2023-10-04T02:07:54.130 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53435.log 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47326.log.gz 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48789.log.gz 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45054.log 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2023-10-04T02:07:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53374.log 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45054.log.gz 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55303.log 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53374.log.gz 2023-10-04T02:07:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46242.log 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55303.log.gz 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52464.log 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46242.log.gz 2023-10-04T02:07:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52222.log 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52464.log.gz 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44439.log 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52222.log.gz 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2023-10-04T02:07:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51038.log 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44439.log.gz 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52252.log 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51038.log.gz 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34324.log 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52252.log.gz 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45693.log 2023-10-04T02:07:54.135 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34324.log.gz 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45693.log.gz 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38810.log 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2023-10-04T02:07:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43740.log 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44682.log 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38810.log.gz 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43792.log 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43740.log.gz 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46448.log 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44682.log.gz 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46022.log 2023-10-04T02:07:54.137 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43792.log.gz 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51369.log 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46448.log.gz 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54533.log 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46022.log.gz 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51369.log.gz 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48426.log 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53960.log 2023-10-04T02:07:54.138 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54533.log.gz 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45027.log 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48426.log.gz 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40019.log 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53960.log.gz 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48057.log 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45027.log.gz 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53666.log 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40019.log.gz 2023-10-04T02:07:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.48057.log: --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48057.log.gz 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53666.log.gz 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46046.log 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54032.log 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2023-10-04T02:07:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50398.log 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46046.log.gz 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47656.log 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54032.log.gz 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.50398.log: --verbose -- /var/log/ceph/ceph-client.admin.49972.log 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50398.log.gz 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52373.log 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47656.log.gz 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51551.log 2023-10-04T02:07:54.141 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49972.log.gz 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50886.log 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52373.log.gz 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55195.log 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51551.log.gz 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50886.log.gz 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2023-10-04T02:07:54.142 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55195.log.gz 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49631.log 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51189.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz -- 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.34276.log 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53495.log 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49631.log.gz 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2023-10-04T02:07:54.143 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34276.log.gz 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53495.log.gz 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48901.log 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52312.log 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49846.log 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48901.log.gz 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55169.log 2023-10-04T02:07:54.144 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52312.log.gz 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56975.log 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49846.log.gz 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42284.log 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55169.log.gz 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54611.log 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56975.log.gz 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48457.log 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42284.log.gz 2023-10-04T02:07:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54611.log.gz 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51459.log 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48457.log.gz 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43911.log 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2023-10-04T02:07:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56814.log 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43911.log.gz 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48304.log 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56814.log.gz 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47140.log 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2023-10-04T02:07:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48304.log.gz 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47140.log.gz 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54954.log 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2023-10-04T02:07:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.54954.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45827.log 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54954.log.gz 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52162.log 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45827.log.gz 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56945.log 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52162.log.gz 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2023-10-04T02:07:54.149 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34359.log 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56945.log.gz 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55749.log 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50275.log 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34359.log.gz 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56014.log 2023-10-04T02:07:54.150 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55749.log.gz 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43111.log 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50275.log.gz 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48871.log 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56014.log.gz 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52680.log 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48871.log.gz 2023-10-04T02:07:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47812.log 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53834.log 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52680.log.gz 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47716.log 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47812.log.gz 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46970.log 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53834.log.gz 2023-10-04T02:07:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40579.log 2023-10-04T02:07:54.153 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47716.log.gz 2023-10-04T02:07:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56360.log 2023-10-04T02:07:54.153 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46970.log.gz 2023-10-04T02:07:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40579.log.gz 2023-10-04T02:07:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56360.log.gz 2023-10-04T02:07:54.154 INFO:teuthology.orchestra.run.smithi027.stderr: 28.7% -- replaced with /var/log/ceph/ceph-client.admin.42652.log.gz 2023-10-04T02:07:54.154 INFO:teuthology.orchestra.run.smithi027.stderr: 26.4% -- replaced with /var/log/ceph/ceph-client.admin.44511.log.gz 2023-10-04T02:07:54.154 INFO:teuthology.orchestra.run.smithi027.stderr: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-04T02:08:10.377 INFO:teuthology.orchestra.run.smithi027.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-04T02:08:15.940 INFO:teuthology.orchestra.run.smithi027.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-04T02:48:06.487 INFO:teuthology.orchestra.run.smithi027.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-04T02:57:59.454 INFO:teuthology.orchestra.run.smithi027.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-04T02:58:01.958 INFO:teuthology.orchestra.run.smithi027.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-04T02:58:01.960 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-10-04T02:58:01.960 INFO:teuthology.orchestra.run.smithi027.stderr:real 50m8.147s 2023-10-04T02:58:01.960 INFO:teuthology.orchestra.run.smithi027.stderr:user 49m5.357s 2023-10-04T02:58:01.960 INFO:teuthology.orchestra.run.smithi027.stderr:sys 2m30.450s 2023-10-04T02:58:01.962 INFO:tasks.ceph:Archiving logs... 2023-10-04T02:58:01.964 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079/remote/smithi027/log 2023-10-04T02:58:01.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-04T02:59:54.462 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-04T02:59:54.474 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-04T02:59:54.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-04T02:59:54.549 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-04T02:59:54.566 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-04T02:59:54.566 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout: remote refid st t when poll reach delay offset jitter 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================== 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout:*hv01.front.sepi 44.4.53.2 2 u 120 256 377 0.142 +0.641 0.157 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout:+hv02.front.sepi 63.231.80.2 3 u 257 256 377 0.185 -2.885 0.162 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout:+hv03.front.sepi 74.6.168.72 3 u 246 256 377 0.175 -2.470 0.147 2023-10-04T02:59:54.633 INFO:teuthology.orchestra.run.smithi027.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-04T02:59:54.634 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-04T02:59:54.642 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-04T02:59:54.643 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-04T02:59:54.653 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-04T02:59:54.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-04T02:59:54.674 INFO:teuthology.task.internal:Duration was 9402.110653 seconds 2023-10-04T02:59:54.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-04T02:59:54.685 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-04T02:59:54.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-04T02:59:55.078 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-04T02:59:55.079 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2023-10-04T02:59:55.081 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-04T02:59:55.124 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-04T02:59:55.125 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-04T02:59:55.276 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-04T02:59:55.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-04T02:59:55.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-04T02:59:55.782 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-04T02:59:55.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-04T02:59:55.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-04T02:59:55.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-04T02:59:55.870 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2023-10-04T02:59:55.903 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-04T02:59:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T02:59:55.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-04T02:59:55.962 INFO:teuthology.task.internal:Transferring archived files... 2023-10-04T02:59:55.963 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079/remote/smithi027 2023-10-04T02:59:55.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-04T02:59:56.015 INFO:teuthology.task.internal:Removing archive directory... 2023-10-04T02:59:56.015 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-04T02:59:56.055 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-04T02:59:56.067 INFO:teuthology.task.internal:Not uploading archives. 2023-10-04T02:59:56.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-04T02:59:56.082 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-04T02:59:56.083 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-04T02:59:56.128 INFO:teuthology.orchestra.run.smithi027.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 4 02:59 /home/ubuntu/cephtest 2023-10-04T02:59:56.128 INFO:teuthology.orchestra.run.smithi027.stdout: 673281 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 4 00:33 /home/ubuntu/cephtest/mnt.0 2023-10-04T02:59:56.129 INFO:teuthology.orchestra.run.smithi027.stdout: 673707 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 4 00:34 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-04T02:59:56.129 INFO:teuthology.orchestra.run.smithi027.stdout: 687589 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 4 00:34 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-04T02:59:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-04T02:59:56.130 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-04T02:59:56.130 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-04T02:59:56.137 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-04T02:59:56.149 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-04T02:59:56.164 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi027 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-04T02:59:56.165 INFO:teuthology.nuke:Checking targets against current locks 2023-10-04T02:59:56.195 DEBUG:teuthology.nuke:shortname: smithi027 2023-10-04T02:59:56.195 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-04T02:59:56.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-04_00:09:14-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7410079', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-04 00:09:38.960880', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF0ecNHL7KfIGkixkKWDByilxad+DFHOUw5+x9jVxBF8gvTOZ1tVfoPG8cDk7ttC44bcMPg9G/SHv/LmT/cUSy8='} 2023-10-04T02:59:56.268 INFO:teuthology.orchestra.console.smithi027:Power off 2023-10-04T02:59:56.269 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-04T02:59:56.290 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2023-10-04T02:59:56.291 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-04T02:59:56.313 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-10-04T03:00:00.314 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-04T03:00:00.423 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-10-04T03:00:04.427 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-04T03:00:04.541 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-10-04T03:00:08.544 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-04T03:00:08.659 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2023-10-04T03:00:08.760 INFO:teuthology.orchestra.console.smithi027:Power off completed 2023-10-04T03:00:08.866 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/replicated-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 9402.110652923584 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi027 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7fa2fe7fd78eee9617aa81ca1e9d3def5c5ca231 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=85a3acddc0494e7881b7e5ac249b3474 status: fail success: false 2023-10-04T03:00:08.867 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-04T03:00:08.945 INFO:teuthology.run:FAIL